journalctl helps turn a failed Linux service into a sequence of events you can investigate. Instead of searching unrelated log files, you can select one service, narrow the time window and compare application messages with the service manager’s response.
The useful question is rarely “What errors are on this machine?” It is “What happened immediately before this service stopped working?” Start with that question and the journal becomes much easier to read. This guide covers investigation on systems using systemd; service configuration, dependencies and restart policies belong in the broader guide to managing Linux services with systemd.
Identify the Service Before Reading Its Logs
Begin with the service’s exact unit name and its current status. Use the status operation in systemctl to identify the loaded unit, its state and any reported exit result. The short log excerpt is a starting point, not the complete incident history.
Record the unit name rather than guessing it from the application’s name. An instantiated service may include an instance identifier, such as worker@reports.service. Investigating a different instance can produce convincing logs for an entirely unrelated problem.
Before changing anything, note when users first noticed the fault, whether it followed a deployment or reboot, and what still works. “The website is down” could mean the web server failed to start, the application behind it stopped responding, or requests never reached the machine. Those are different investigations.
Keep the distinction between service state and application health. A running process does not prove that requests succeed. Equally, an intentionally completed task should not be treated as a failed background service simply because it is no longer running.
For a production incident, preserve the initial state before restarting. A restart may restore availability, but it also changes the evidence you are about to inspect.
Build a Small, Useful Journal View
For an initial inspection of an Nginx service, use journalctl with the unit argument --unit=nginx.service, the current boot selector --boot and a recent entry count such as --lines=100. These are arguments to combine, not separate troubleshooting stages.
| Argument | Purpose |
|---|---|
--unit=nginx.service |
Select messages from and about this unit. |
--boot |
Restrict the view to the current boot. |
--lines=100 |
Show the most recent 100 matching entries. |
--reverse |
Display newest entries first. |
--no-pager |
Print without opening an interactive pager. |
The unit selector includes relevant service manager messages, not just output emitted by the application. The official journalctl option reference documents these selectors and their combinations.
Read a small recent sample to find the incident, then expand around it. If the first visible entry is already a failure summary, you have probably started too late. Prefer chronological order when reconstructing a startup sequence; newest first is better for finding the latest event than explaining its cause.
Choose the Right Time Window and Boot
Use --since and --until to bracket an incident. For a hypothetical outage on October 23, 2026, the argument --since="2026-10-23 14:00:00" establishes the beginning of the window. Pair it with an end time after the observed failure.
Leave some room on either side. A failure reported at 14:05 might have started with a configuration reload at 14:02 or a dependency disconnect several minutes earlier. A window that contains only the final error can hide the event that matters.
After a reboot, inspect --list-boots before selecting older records. For the local machine’s journal, --boot=-1 selects the preceding recorded boot. Check its timestamps rather than assuming it contains the outage. Red Hat’s journal troubleshooting documentation covers time windows, boot selection and live log inspection.
Write the timezone into your incident notes. If a monitoring alert, customer report and server log use different timezones, align them before comparing events. Otherwise, a perfectly reasonable search can inspect the wrong hour.
Also consider the scope of the failure. If the whole computer stopped booting, service logs are only part of the evidence. Do not spend an hour investigating an application that never had a chance to start.
Use Priority and Text Searches Without Losing Context
The argument --priority=err selects errors and more urgent priorities. It does not mean “messages containing the word error.” The argument --grep=timeout searches message text using a regular expression. These filters answer different questions.
There is another common trap: -x adds available message catalogue explanations, while -e opens the pager at the end. Neither option restricts output to errors. These behaviours are documented in the Ubuntu journalctl manual.
Use priority filtering to find a candidate event, then remove it while retaining the unit and time window. The surrounding informational messages may identify the configuration file, connection attempt or startup stage that preceded the error.
Do the same with text searches. Searching for “timeout” can locate an incident quickly, but the explanation may appear in a nearby message without that word. Treat search results as pointers into the timeline, not the whole timeline.
A severity label also needs interpretation. Ask whether the entry matches the affected service, occurred during the incident and describes a condition that could explain the symptom. An unrelated warning from the morning is not evidence for an afternoon outage just because it looks alarming.
Find the First Actionable Failure
Read backwards from the final failure summary until you reach the first message that suggests something you can verify. “Service failed” names the outcome. A rejected configuration setting, inaccessible directory or failed connection gives you a testable explanation.
Distinguish startup preparation from application failure
Systemd can report failures before the configured application starts. A result containing 203/EXEC points to an execution failure; 200/CHDIR indicates failure to enter the requested working directory. A result containing 217/USER concerns user credential setup or user namespacing. The systemd execution environment reference defines these service manager exit codes.
Use those results to choose the next inspection. For an execution failure, verify the configured executable and whether the service can access it. For a working directory failure, inspect the requested directory. Do not begin by rewriting application settings when the application has not started.
Follow the path named in the error
If the journal names a file or directory, check that exact path and the account the service uses. Avoid broad permission changes as an experiment. Making an entire directory writable may replace one fault with a security problem and still leave the original explanation unclear.
The guide to Linux file permissions and ownership covers the access checks involved. Keep the investigation narrow: which operation failed, which account attempted it, and which path was involved?
Separate the cause from the restart loop
Repeated startup failures can bury the earliest useful message beneath nearly identical attempts. Group the messages into individual starts and compare the first failure in each. Restarting a service ten times is not ten different tests.
If an application reports a connection problem, identify the destination before changing local settings. Is it connecting to a local database, a remote API or another service on the same machine? Verify that dependency rather than treating every connection error as a network configuration fault.
When the evidence points to memory pressure, storage exhaustion or heavy system load, move to Linux resource monitoring. A log entry provides a timestamp and a symptom; resource measurements help test the explanation.
Watch One Controlled Reproduction
Use --follow with the unit selector to watch matching entries arrive. Keep that view open while performing one approved action in another terminal or application: a configuration validation, a request that triggers the fault, or a planned service restart.
Write down the action’s time and expected result first. Then compare the new messages with the earlier failure. This keeps the test focused and avoids confusing your own troubleshooting activity with the original incident.
For example, suppose a website serves static content but fails on requests that need a database. Reproduce one affected request and inspect that interval, rather than repeatedly restarting the web server. The distinction between the web server and the application behind it is covered in the guide to hosting a website with Nginx and HTTPS.
Be more cautious with remote access services. Before restarting SSH or changing its configuration, keep a working session open and confirm that you have a recovery route. Follow the access precautions in securing SSH access.
After each test, decide what the result rules out. If the evidence has not changed, repeating the same action without a new hypothesis adds noise rather than progress.
Troubleshoot Missing or Incomplete Journal Entries
An empty result is not proof that nothing happened. Start by removing filters one at a time. Check the unit name, widen the time window and confirm the boot selection. Keep a note of which change makes entries appear.
Check access and service scope
For a system service, confirm that your account has permission to read the system journal. For a service managed by your own user instance, investigate the user scope rather than assuming it belongs to the system manager. Do not switch accounts casually during the investigation.
If an authorized elevated inspection reveals records that were previously hidden, address journal access through the machine’s normal administrative policy. Avoid changing journal file permissions by hand. The guide to Linux users, groups and sudo access covers the wider access decision.
Check storage and message retention
Volatile journal storage under /run/log/journal does not survive reboot. Persistent storage normally uses /var/log/journal. With Storage=auto, the presence of the persistent journal directory controls the storage mode. Inspect the machine’s effective configuration rather than assuming a default.
Message rate limits can also drop entries during a logging burst. Inspect suppression notices and the settings RateLimitIntervalSec and RateLimitBurst. Storage behaviour, local configuration overrides and rate controls are defined in the journald configuration reference.
Changing retention now cannot recover entries that were never stored or have already been removed. Configure adequate retention before the next incident, and avoid disabling rate controls simply to make a suppression notice disappear.
Check the application’s own logging destination
Some applications keep their detailed diagnostic logs separately. If the journal shows startup and shutdown messages but little application detail, inspect the application’s configured log destination. Do not assume every request, traceback or authentication event must appear in the service journal.
Preserve Evidence Before Cleaning Up
Once you have isolated the incident, save a bounded excerpt containing the failure and enough surrounding context to explain it. Include the unit name, boot identifier, timezone, relevant configuration change and the action that reproduced the problem.
For readable output, --output=short-iso-precise retains detailed timestamps. The verbose output format exposes stored fields, while json is useful for structured processing. Avoid message only output when the timestamps and identity fields are needed to reconstruct events.
Before sharing logs, inspect them for credentials, tokens, private addresses, usernames and customer data. Keep an access controlled original when appropriate, and share a redacted copy rather than altering the only evidence.
Disk cleanup is a separate task. The argument --disk-usage includes active and archived journal files, while vacuum operations remove archived files. Rotation makes active files eligible for later removal, so combining rotation and vacuuming can delete more history than expected. The Debian journalctl manual documents output formats and these retention operations.
Do not vacuum the journal as a routine first response to a service failure. Preserve useful records first. Where incident evidence must survive loss of the machine, include it in an appropriate offsite backup and recovery process.
Verify Recovery Against the Original Symptom
After applying a justified correction, repeat the same observation that exposed the failure. Check the service state, inspect the new startup messages and test the application through the route users actually depend on.
For a web service, that means checking a real request, not just confirming that a process exists. For a scheduled job, inspect its completed work. For a dependency failure, confirm that the connection succeeds and the intended operation finishes.
If the service now works locally but remains unreachable remotely, treat that as a new boundary to investigate. Review the listening address, access path and Linux firewall configuration rather than continuing to restart a healthy application.
Keep the final incident note short: what failed, which journal entries supported the diagnosis, what changed and how recovery was verified. A useful journal investigation ends with evidence that explains the fault, not just a terminal that has stopped printing red text.