Managing Linux Services with systemd

Managing Linux services with systemd means controlling two separate things: what runs now, and what should start automatically later. Confuse those jobs and a service can work perfectly until the next reboot, or keep running after you thought you had disabled it.

The main administration tool is systemctl. It lets you inspect services, change their running state, manage startup settings and adjust how systemd supervises applications. This guide covers those tasks, along with service overrides, dependencies and a practical troubleshooting routine. If you are still preparing the machine, begin with setting up your first Linux server before changing services that provide remote access or network connectivity.

How systemd services and units fit together

Systemd manages objects called units. A service is one kind of unit, not another name for everything systemd controls. Service unit names end in .service, such as nginx.service. Other unit types include sockets for connection handling, timers for scheduled activation, mounts for filesystems and targets for grouping related units.

This distinction matters during maintenance. An application might have both a service unit and a socket unit. Stopping its process does not necessarily remove the mechanism that starts it again. A scheduled job may depend on a timer rather than a permanently running service.

A unit definition tells systemd how to manage an object. For a service, that can include the executable, account, working directory, startup behavior and restart policy. The application usually has its own configuration too. Changing a web server’s settings and changing its systemd unit are different operations.

Before following systemd instructions, confirm that systemd actually manages the environment you are administering. Check the identity of process 1 using a process viewer. A container, recovery environment or Linux installation using another init system may need a different approach. A familiar shell prompt does not guarantee a familiar service manager.

Inspect services before changing them

Start with discovery rather than guessing a unit name. The list-units action shows units loaded into the manager, while list-unit-files shows installed unit files and their enablement states. The --type=service filter narrows either view to services; --all includes inactive loaded units, and --failed helps identify failures. These distinctions appear in the SUSE administration guide to systemd service inspection.

Do not assume the package name, executable name and unit name match. Find the installed unit, read its description and confirm that it belongs to the application you intend to change. Keep the full suffix when recording maintenance steps: example.service and example.socket are different objects.

Read status as several separate signals

The status action provides a useful starting point. Read the load information, active state, process details and recent messages together. A loaded definition does not prove that its application started successfully, and a running process does not prove that clients can use it.

For a persistent server application, active (running) is usually the expected state. A task that performs initialization and exits may legitimately show active (exited). An inactive unit is not automatically broken: it may be waiting for activation or may have completed its work.

Write down the current running state and startup setting before making changes. That small record makes rollback much less dependent on memory, particularly when a maintenance window starts to get untidy.

Use an authorized account for changes to system services. Avoid doing an entire investigation inside a root shell when only the final action needs elevated privileges. The guide to Linux users, groups and sudo access covers how to separate routine inspection from administrative control.

Starting a service is not enabling it

Runtime control and automatic activation are separate. Starting a service activates it now. Enabling it connects it to the activation relationships defined by its installation settings, commonly startup during boot. Neither operation substitutes for the other.

Common systemctl actions and their purpose
Action or option Purpose Maintenance consideration
start Activate the unit now. Does not enable future automatic startup.
stop Deactivate the unit now. Does not remove its startup settings.
restart Stop and start the service. Also starts an inactive service.
reload Request an application configuration reload. Requires support from the service.
enable Apply the unit’s automatic activation settings. Does not start it immediately by default.
disable Remove enablement links. Does not stop an already running service by default.
--now Add immediate runtime action to enablement changes. With enable, start now; with disable, stop now.

The Red Hat documentation on managing system services covers these runtime and startup operations. Treat them as separate decisions even when you combine them into one administrative step.

Consider a new web server. During testing, you might start it without enabling automatic startup. Once its configuration and external access have been checked, enabling it becomes appropriate. Before retiring that server, you would consider both stopping the application and removing its automatic activation.

For the application setup rather than its service lifecycle, use the separate guide to hosting a website with Nginx and HTTPS. A successful service start is only one part of a working website.

Disabling and masking solve different problems

Disabling a unit does not prohibit every future start. An administrator, dependency or activation mechanism may still start it. Masking is stronger: it blocks activation of the masked unit until the mask is removed. It is an administrative restriction, not a substitute for uninstalling software or controlling who can administer the machine.

Use masking only when you have a reason to block activation, and record that reason. Do not unmask an unfamiliar service just because a start attempt fails. The restriction may be deliberate. When checking why an application returns after being stopped, inspect associated sockets, timers and other triggers too.

Choose between reload, restart and daemon-reload

These operations affect different layers. A service reload asks the application to reread its configuration. A restart replaces its running processes. A manager reload makes systemd reread unit definitions; it does not, by itself, restart the applications those definitions describe.

The action for that manager operation is daemon-reload. Use it after manually changing unit definitions or drop-in files. It is not the correct response to every application configuration change. The upstream systemctl reference for reloads, masking and unit control distinguishes these operations.

Before reloading an application, use its configuration validation facility where available. Then check whether the changed setting supports reload. Some settings require a restart even when the application supports reloading other parts of its configuration.

Do not assume reload means zero disruption. Its behavior belongs to the application. Plan verification around the service users receive, not just whether the administration tool returns successfully.

Software updates deserve the same care. Reloading configuration does not necessarily replace an old executable or loaded library. Keep service maintenance aligned with your approach to Linux security updates and automatic patching, including any required process restarts.

Troubleshoot failed services without restarting blindly

A failed start is evidence. Preserve it before changing several settings or clearing recorded state. Note the unit name, failure time, exit information and last relevant messages. Then identify whether the failure occurred before the application launched, during initialization or after it began accepting work.

The status display is a summary, not a complete investigation. Use the journal to inspect the affected unit around the failure time. Read messages before the final error as well as after it. “Failed to start” reports the outcome; an earlier message may identify a missing file or rejected setting.

For filtering by unit, boot and time range, continue with troubleshooting Linux services with journalctl. Keep that investigation focused rather than collecting pages of unrelated messages.

Separate application failures from execution failures

If the application reports an invalid configuration value, investigate that value. If systemd cannot launch the executable, check the configured path, account and access permissions. Those are different failure paths and need different fixes.

A useful first question is whether anything changed: a package upgrade, moved directory, rotated credential, edited override or altered mount. Compare the failure time with your change records. Change one suspected cause at a time so that a successful recovery tells you something.

If the unit cannot be found, confirm that the required package is installed and that you are using the correct unit name and manager scope. A unit search cannot repair an absent installation. The guide to APT, DNF and Pacman package management covers that separate layer.

Check the operating conditions too

Do not overlook storage capacity, memory pressure or unavailable dependencies. An application may be configured correctly but lack the resources needed to initialize. Compare its messages with CPU, memory, disk and network monitoring before changing startup timeouts or restart behavior.

If repeated failures have reached a start limit, correct the fault before trying again. The reset-failed action clears recorded failure state and start-rate counters; it does not repair the cause. Clearing a red status indicator is not the same as restoring service.

After recovery, test one useful operation through the application. For a database, that could be an authenticated query. For a website, it could be a request that reaches the backend rather than a static holding page. Keep the test small, repeatable and representative.

Use drop-in overrides for local changes

For routine adjustments, prefer a drop-in override to editing a package’s unit file directly. Drop-ins let you change selected settings while retaining the rest of the vendor definition. A complete local replacement can be appropriate, but it also gives you more configuration to maintain.

The edit action opens an override for the selected unit. System service overrides commonly live beneath /etc/systemd/system/, in a directory named after the unit. The Oracle Linux documentation on service unit overrides describes this arrangement and the distinction between an override and a replacement unit.

Before editing, inspect the original definition and existing drop-ins. Use the cat action to review the backing files. Otherwise, you may add a setting that conflicts with a previous administrator’s change or misunderstand which file supplies the current behavior.

Some settings replace earlier values; others accumulate entries and require an explicit reset before replacement. Check the relevant directive documentation rather than assuming that the last visible line always wins.

After saving an override through the normal edit workflow, systemd reloads its configuration. Files changed manually require a manager reload. Neither path guarantees that a running application has adopted settings that only apply when its processes start.

Make one change, apply the required runtime action and test it. Record the reason for the override alongside the maintenance record. “Increase startup allowance because recovery takes longer on this storage” is useful. “Fix service” leaves the next administrator with a small mystery.

For rollback, remove or revise the override you introduced, reload the manager when required and repeat the application test. Avoid broad cleanup actions unless you intend to remove all affected local customizations.

Define the execution environment explicitly

A program that works in your terminal can still fail as a service. Your shell session may provide a working directory, environment variables and account permissions that the service does not have. Treat the service environment as something to define, not something inherited from whichever terminal last worked.

For a custom service, decide which account should run it, where its files belong, which executable it should use and where it needs write access. Prefer a dedicated account for an application that does not need administrative privileges.

Review directory access as well as file access. Before weakening permissions to make a failure disappear, work through Linux file permissions and ownership. Granting everyone write access is a fast way to replace one problem with several less obvious ones.

The User, WorkingDirectory and EnvironmentFile directives address parts of the execution environment. Environment variables are not a suitable general store for service secrets. Systemd offers credential mechanisms for supported applications, while restrictions such as NoNewPrivileges can reduce what service processes may do. These behaviors are documented in the Ubuntu manual for systemd execution environments and security settings.

Apply restrictions gradually. Start from the application’s documented requirements, test normal operation, then test maintenance tasks such as writing state or replacing temporary files. A service that starts successfully but fails during its first scheduled task is not ready.

Keep unit files concise. They should describe how systemd runs the application, not become an improvised deployment system containing unrelated setup work. Application installation, data migration and service supervision deserve separate, understandable steps.

Set restart behavior without hiding faults

For a persistent custom application, automatic restart can recover from a process failure. The Restart directive controls the policy, and RestartSec introduces a delay. The on-failure policy is often appropriate for a service expected to remain running, but it is not a universal choice for scheduled or one-time tasks.

Systemd’s service type also affects what startup completion means. A process being launched is not necessarily an application being ready for requests. Choose the type supported by the application rather than selecting one because it suppresses an error. The upstream service configuration manual documents startup types, restart policies and their interaction with start limits.

Decide what should happen after repeated failures. A worker processing disposable jobs may tolerate retries differently from a task that sends payments or modifies records. Restart supervision does not make an application’s work safe to repeat.

Watch for restart loops. A service that stays alive for a few seconds between crashes is not healthy, even if it repeatedly appears active during casual checks. Keep recovery policy separate from alerting: the machine may restart the process, but someone still needs to investigate recurring failure.

Handle dependencies and startup ordering separately

Dependencies describe relationships between units; ordering describes sequence. The Wants directive pulls in another unit without making its successful activation a strict prerequisite. Requires creates a stronger relationship. Neither replaces the ordering expressed through After or Before.

An ordering rule alone does not start the named dependency. Nor does successful activation necessarily prove that another application is ready to serve requests. These distinctions are covered in the Debian manual for systemd dependencies and ordering.

Keep local dependency changes small. Before adding a relationship, write down the failure it should prevent and how you will test it. Avoid adding fixed delays simply because a service sometimes loses a startup race. A delay can hide the race until a slower boot exposes it again.

Network targets are not connectivity tests

The network.target unit does not guarantee a configured connection. The meaning of network-online.target depends on the network manager’s readiness criteria, and it does not continuously verify connectivity after boot. The systemd documentation on network startup synchronization explains why these targets should not be added indiscriminately.

For an application that contacts a remote dependency, plan for that dependency to become unavailable after startup too. Check its retry and reconnection behavior. Getting the boot order right cannot prevent an outage ten minutes later.

Distinguish system services from user services

System services and user services belong to separate manager scopes. The --user option selects the current user’s manager. If an installed service seems to have vanished, check the intended scope before reinstalling software or creating another unit with the same name.

A system service can still run under an unprivileged account; “system service” does not mean “must run as root.” Choose scope based on who owns the workload and when it needs to exist.

For user services that must run without an active login, lingering may be relevant. Enabling it causes the user manager to start at boot and remain after logout, as documented in the Debian loginctl manual on user lingering. Make that an intentional administration decision rather than enabling it for every account.

This distinction is useful for personal development tools and rootless workloads. Keep application lifecycle decisions consistent with the setup described in local development with Docker and Podman, rather than creating overlapping managers for the same process.

Follow a repeatable service maintenance routine

Good service administration is mostly controlled change. The following sequence works for a configuration adjustment, package update or local override without treating every task as an emergency:

  1. Identify the unit and scope. Confirm the service, its activation triggers and the manager responsible for it.
  2. Record the starting state. Note running status, enablement, existing overrides and a useful application test.
  3. Prepare recovery. Preserve the configuration you will change and decide how to restore it.
  4. Make one change. Validate it, then choose the required reload, restart or manager reload.
  5. Verify the result. Check status, relevant messages and the application test, then document the outcome.

Take extra care with SSH, networking and anything required to reach the server. Keep an existing session open, arrange console access where possible and confirm a fresh connection before ending maintenance. The guide to securing SSH access covers the access controls around that service.

Configuration recovery and data recovery are different preparations. A saved unit override will not restore a damaged database. Match the workload to an appropriate offsite backup and recovery testing plan before making changes that could affect stored data.

Finish by checking the intended future state, not just the current process. If automatic startup changed, verify it during a planned reboot when access and recovery are available. A service that works now is useful; a service whose behavior you can predict after maintenance is manageable.