How to monitor a Linux server with an agent

You monitor a Linux server by installing an agent that regularly measures CPU, memory, swap and disks and raises an alert when a threshold is crossed. With inping that takes a single shell command. The agent only reads, opens no port and is visible in inping within a minute of starting.

What do you see afterwards?

• Under Agents: hostname and system, status Online, Last seen and the gauges for CPU, memory and disks.
• The agent's overview: the history of CPU, memory and disks, along with version, architecture and cores.
• Accounts: every account on the host with the CPU share and memory of its processes, from the second measurement on. On a shared host this shows you which customer is generating the load.
• Packages: the installed packages from dpkg, rpm, apk or pacman.
• Jobs: cron entries and systemd timers with schedule, command and user.
• Project → Packages: runtimes and packages of the projects on the server, read from the lock files (on the first run after the start, then checked every 15 minutes).
• Vulnerabilities: known vulnerabilities in these packages appear on the project and on the agent within a day.

Which metrics matter on a Linux server?

Four values explain most problems. The limits mentioned here are starting points to adapt to the server: a database server may fill its memory, a build server may max out its CPU.

CPU usage

The share of time the processors are working. Short spikes are normal and say little. What matters is high usage over several minutes, which is why the default in inping only applies after 5 minutes above 90%.

Load average

The number of processes running or waiting, averaged over 1, 5 and 15 minutes. On Linux this includes processes waiting for the disk. Always read the value relative to the number of cores: a load of 4 is relaxed on 8 cores and too much on 2. High load with low CPU usage often points to slow disks. There is no default rule for load, as the right value depends on the cores. You create a rule on Load (5 min) yourself.

Memory and swap

Linux uses free memory as a file cache, so "little free" is no cause for alarm. inping calculates total minus available, which matches the available column of free -h: the cache does not count as used. When memory really does run short, the system starts to swap and in the extreme ends processes (the OOM killer). Growing swap with high memory use is the earlier warning sign. If you need it, create your own rule for this, for example swap above 50% for 15 minutes.

Disks

A full disk reliably brings services to a halt: databases can no longer write, logs get lost. inping therefore warns per mount immediately from 90%. Work out what that means: on 40 GB it leaves 4 GB, on 2 TB still 200 GB. On small disks a second rule with a lower value is worthwhile, on large ones a rule on the used space in bytes. Do not set the limit too high: ext4 reserves 5% for root by default, so services without root run out of space before 100%. inping does not measure inodes; check those with df -i if needed.

Requirements

• a free inping account, and a project if the agent is to report packages
• curl or wget, plus sudo or a root shell
• outgoing HTTPS to www.inping.io on port 443; nothing needs to be opened inbound
• amd64 or arm64; the script picks the matching build
• running time synchronisation (chronyd, systemd-timesyncd, ntpd): reports more than 60 seconds ahead of the platform's clock are rejected
• a kernel from 5.13 for the agent's Landlock sandbox; older kernels still run the agent, then with the systemd hardening only

Step by step on Debian and Ubuntu

These steps also apply to Raspberry Pi OS, Linux Mint and other Debian-based systems with systemd.

1. Create the agent and copy the command

In inping under Agents → Add agent or in the project under Servers → Add agent. The dialog shows the registration token exactly once, and it is valid for 24 hours. It also shows the finished command for every system, with the token, address and project ID already filled in. One click copies it. All systems and downloads are also listed in the app under Agents → Installation and downloads (/agents/install).

2. Run the command on the server

curl -fsSL https://www.inping.io/install.sh | INPING_TOKEN=<token> sudo --preserve-env=INPING_TOKEN sh -s -- --project <project-id>=/srv/shop

<token> and <project-id> are placeholders here. Repeat --project for each project on the server; the ID is under Project → Settings. The script downloads the agent, checks the SHA-256 sum (and the signature if minisign is installed), creates the user inping-agent, writes the configuration and starts the service. The last line reads inping: inping-agent <version> installed.

3. Give read access to the project

So that the agent can read composer.lock, package-lock.json and similar files, add the service user to the group that owns the project. Never run the agent as root.

sudo usermod -aG www-data inping-agent
sudo systemctl restart inping-agent

Installing without root

For a shared host or a customer's server without root access: the same script with --user, run under the account that may read the project.

curl -fsSL https://www.inping.io/install.sh | INPING_TOKEN=<token> sh -s -- --user --project <project-id>=$HOME/www/shop

Everything stays in the home directory, and the agent reads exactly what your account may read. Check with systemctl --user status inping-agent. Two limitations: the systemd user manager stops at the last logout unless lingering is active (sudo loginctl enable-linger <account>; the script tries this itself and tells you if it does not work). And the sandbox is thinner than with the system service, although Landlock still applies from kernel 5.13.

Network per account (optional)

Data received and sent per account is counted by systemd's IP accounting. It is switched on with --ip-accounting, and the agent only reads the counters:

curl -fsSL https://www.inping.io/install.sh | sudo sh -s -- --ip-accounting

This is a system-wide setting. Look at its effect on a heavily loaded host before you roll it out everywhere; --no-ip-accounting reverses it.

Specifics on Rocky Linux, AlmaLinux, CentOS Stream and Fedora

Everything above applies, and the same command works:

curl -fsSL https://www.inping.io/install.sh | INPING_TOKEN=<token> sudo --preserve-env=INPING_TOKEN sh -s -- --project <project-id>=/var/www/shop

The web server's group here is usually apache or nginx:

sudo usermod -aG nginx inping-agent && sudo systemctl restart inping-agent

• SELinux can stay on enforcing, and no policy module is needed. If ausearch -m avc -ts recent still shows denials for inping-agent, label the project directory for reading rather than switching SELinux off.
• firewalld allows outgoing connections by default.
• Vulnerabilities: packages from Rocky Linux and AlmaLinux are checked against security advisories. Fedora reports no ecosystem, so its packages are only listed and not checked.

Specifics on Alpine Linux

Alpine has no systemd and no useradd. The script installs and registers the agent; you set up the service yourself with OpenRC. Requirements: apk add curl and root. First create the user, then the script sets up the agent:

adduser -S -D -H -h /var/lib/inping-agent -s /sbin/nologin inping-agent
curl -fsSL https://www.inping.io/install.sh | INPING_TOKEN=<token> sh -s -- --project <project-id>=/srv/shop

Then create and start the OpenRC service:

cat >/etc/init.d/inping-agent <<'EOF'
#!/sbin/openrc-run
name="inping-agent"
description="inping agent: host metrics and packages for inping"
supervisor=supervise-daemon
command="/usr/local/bin/inping-agent"
command_args="run --config /etc/inping/agent.toml"
command_user="inping-agent:inping-agent"
respawn_delay=10
output_log="/var/log/inping-agent.log"
error_log="/var/log/inping-agent.log"

depend() {
	need net
	after firewall
}
EOF
chmod 0755 /etc/init.d/inping-agent
touch /var/log/inping-agent.log && chown inping-agent /var/log/inping-agent.log
rc-update add inping-agent default
rc-service inping-agent start

Read access comes from addgroup inping-agent www-data (or the project's group), followed by rc-service inping-agent restart. This is how you check it:

rc-service inping-agent status      # started
tail -n 20 /var/log/inping-agent.log

Update: the built-in updater restarts the service through systemctl and therefore cannot be used on Alpine. Run the script again without a token instead; the configuration and registration are kept:

curl -fsSL https://www.inping.io/install.sh | sh && rc-service inping-agent restart

Uninstall:

rc-service inping-agent stop && rc-update del inping-agent default && rm /etc/init.d/inping-agent
curl -fsSL https://www.inping.io/install.sh | sh -s -- --purge

The user remains; deluser inping-agent removes it.

How do you check that it works?

systemctl status inping-agent          # active (running)
journalctl -u inping-agent -n 20       # the lines below
sudo inping-agent check                # the configuration from the platform (interval, metrics, projects)

The log of a healthy start looks like this:

level=INFO msg=sandbox enforced=true detail=…
level=INFO msg=starting version=1.2.0 platform=https://www.inping.io
level=INFO msg=configuration interval_seconds=60 buffer_hours=24 metrics="[cpu memory disk network containers users platform]"
level=INFO msg="host packages reported" packages=612
level=INFO msg="host jobs reported" jobs=9
level=INFO msg="inventory reported" project=01m5qxbbej7tdq8m4hhqvr9vhn components=84

In inping the agent is Online within a minute, the host packages follow a few seconds after the start, and the project's packages at the first check. The history starts from the second measurement.

Thresholds and alerts

Every new agent starts with four rules that you can change, switch off or delete:

You create your own rules on the agent's page under Settings → Alert rules: a metric is above, below or equal to a value, for a number of minutes (0 to 1440). An agent can have up to 50 rules, including several on the same metric. An alert opens once and closes once, when the value is back in range.

On top of that comes the offline alert: after three missed reports the agent counts as offline. For you to hear about it, the project the agent is assigned to needs a notification rule for the event, for example Server threshold exceeded or Agent offline, by mail or webhook.

To test it: create a rule CPU above 1% for 0 minutes. The alert opens with the next report. Delete the rule afterwards and the alert closes again.

What may the agent do on the server?

• It runs as its own user without capabilities, with ProtectSystem=strict, NoNewPrivileges and a syscall filter. systemd-analyze security rates the service at 1.3.
• From kernel 5.13 it also locks itself down with Landlock at startup: reading everywhere, writing only in /var/lib/inping-agent.
• The configuration /etc/inping/agent.toml belongs to root. Which project directories it reads is set there, not in the platform.
• The script adds the user to the groups systemd-journal and adm, if present, so that log sources set in inping can be read. Log sources are off until you create one.
• Updates run separately from the service as root and only install releases whose signature matches the key built into the program.
• Disk usage per account is measured by a separate timer. It is the only part with a capability (CAP_DAC_READ_SEARCH), reads only metadata, never file contents, and has no network.

Update

The update runs automatically: every 6 hours the timer runs inping-agent update, installs a newer signed version, restarts the service and rolls back if it does not come up. By hand:

sudo inping-agent update --check   # only look
sudo inping-agent update           # install now

The install command without a token also updates, keeping the configuration and registration.

Uninstall

curl -fsSL https://www.inping.io/install.sh | sudo sh -s -- --uninstall   # service and binary
curl -fsSL https://www.inping.io/install.sh | sudo sh -s -- --purge       # also configuration, state and user

Then delete the agent in inping under Agents → (agent) → Settings → Delete agent. Its metrics and alerts go with it. For an installation without root, add --user --uninstall or --user --purge to the install command.

When nothing arrives

• cannot start or registration failed in the log: the token has already been used, has expired or belongs to a different platform. Create a new one under New registration token and run the command again with it.
• configuration unusable: a typo in /etc/inping/agent.toml. Unknown keys are an error on purpose.
• The agent runs but stays Not registered yet or Offline: outgoing HTTPS is blocked, api_url points to the wrong host, or the clock is more than a minute fast. Test the connection as the agent's user:

curl -sI https://www.inping.io/api/v1/releases/agent/latest

• Metrics arrive, packages do not: the following command shows what the service user can read. A directory it may not enter is reported as "nothing".

sudo -u inping-agent inping-agent inventory --config /etc/inping/agent.toml

• Host packages are missing: the host has neither dpkg, rpm, apk, Homebrew nor pacman, or the configuration contains [host] packages = false.

Frequently asked questions

Further reading

• All the ways to connect servers and websites to inping
• Monitor a Windows server
• Monitor Docker containers
• GDPR server monitoring
• Plans and limits for agents
• Full installation guide for all systems