Incus OS: backup and monitoring without a shell

Read this page in French

On a classic server, you back up and monitor with an installed agent and SSH access. On Incus OS there is neither shell nor third-party agent: everything goes through the API and the trusted client. This guide shows how to back up instances (snapshots, exports), back up the node itself (recovery keys), and monitor an Incus OS cluster with an external Prometheus and centralised logs. Intended audience: administrators running Incus OS in production, not only in a lab.

What you will learn

• Back up instances with snapshots and portable exports.
• Back up the node: retrieve its encryption keys.
• Point an external Prometheus at the Incus metrics.
• Centralise logs towards a remote syslog server.
• Watch the health of the cluster through the API.

Prerequisites

• A running Incus OS cluster. See Incus OS without a shell.
• A trusted Incus client with a remote to one member (node1:).
• A backup server or a Prometheus reachable from the administration network.

The shell-less model

The absence of a shell closes the classic habits: no ssh to launch a dump, no node_exporter dropped by hand onto the node. That is not an oversight, it is the model of Incus OS. Everything below is driven remotely, from the administration workstation, through the Incus API and the system API prefixed with /os. Nothing is installed on the node.

Backing up instances

The quickest backup is the snapshot: a frozen copy of the instance, kept in the pool, useful before a risky operation. You create and list it remotely:

incus snapshot create node1:app1 snap-before-update
incus snapshot list node1:app1

+--------------------+-----------------------+------------+----------+
|        NAME        |       TAKEN AT        | EXPIRES AT | STATEFUL |
+--------------------+-----------------------+------------+----------+
| snap-before-update | 2026/07/03 09:27 CEST |            | NO       |
+--------------------+-----------------------+------------+----------+

A snapshot stays on the node: it protects against a mistake, not against losing the node. For an offsite backup, the export produces a portable archive you can store anywhere:

incus export node1:app1 /path/app1.tar.gz --instance-only

The archive holds the instance and its snapshots, or the instance alone with --instance-only. You restore it onto any member of the cluster, or onto another cluster, with the reverse operation:

incus import node1:/path/app1.tar.gz

Automate from the administration host

Since everything goes through the client, a plain cron job on the trusted administration workstation is enough to schedule snapshots and exports. The Incus OS node has nothing to run or host for that.

Backing up the node itself

Backing up instances is not enough: an Incus OS node encrypts its disks (root, swap, pool), and without the recovery keys, a surviving disk stays unreadable. The system API exposes a dedicated backup that gathers those secrets:

incus query -X POST node1:/os/1.0/system/:backup > node1-backup.tar.gz
tar tzf node1-backup.tar.gz

recovery.root.key
recovery.swap.key
state.txt
zpool.local.key

The archive contains the decryption keys (recovery.root.key, recovery.swap.key, zpool.local.key) and a state.txt describing the system state. It is short but critical: keep it in a vault, off the node, for every cluster member. Restoration goes through the symmetric /os/1.0/system/:restore endpoint.

A backup you have never restored is a hypothesis, not a backup. Since this archive exists solely to make an encrypted disk readable again, the only test that proves anything is decrypting with it, on a spare machine, before you need it for real.

These keys unlock the node

The contents of this backup allow the node's disks to be decrypted. Treat it as a secret of the highest sensitivity: encrypted storage, restricted access, never in a Git repository or an open share.

Monitoring with an external Prometheus

Incus exposes its metrics in OpenMetrics format on the /1.0/metrics endpoint, served over HTTPS with certificate authentication. An external Prometheus collects them without installing anything on the node:

curl -sk --cert client.crt --key client.key \
  https://192.168.10.131:8443/1.0/metrics

# HELP incus_cpu_seconds_total The total number of CPU time used in seconds.
# TYPE incus_cpu_seconds_total counter
incus_cpu_seconds_total{cpu="0",mode="user",name="app1",project="default",type="container"} 0.84
# HELP incus_disk_read_bytes_total The total number of bytes read.
# TYPE incus_disk_read_bytes_total counter
incus_disk_read_bytes_total{device="sda",name="app1",project="default",type="container"} 28672

Every series carries the instance name, its project and its type, which lets you follow CPU, disk, memory and network per workload. On the Prometheus side, configure a job pointing at each member with the certificate in tls_config.

A certificate limited to metrics

Rather than the administration certificate, recent Incus versions accept a metrics-only certificate, with no management rights. Use it for Prometheus scraping: if that certificate leaks, it exposes only read-only measurements, not control of the cluster.

Centralising logs

With no shell, you do not read journalctl on the node. The system API lets you forward logs to a remote syslog server. The endpoint exposes the expected configuration:

incus query node1:/os/1.0/system/logging

{
  "config": {
    "syslog": {
      "address": "",
      "log_format": "",
      "protocol": ""
    }
  }
}

By filling in the collector address, the protocol and the format, each node pushes its journals to your centralised logging stack, a syslog server or a collection pipeline. That is the logical counterpart of the shell-less model: the node does not archive locally, it emits outwards.

Watching cluster health

Beyond metrics, two quick checkpoints tell you about the general state. The root of the system API says whether a node has finished booting and stands ready:

incus query node1:/os/1.0

A system_is_ready field at true confirms it is operational. At cluster level, incus cluster list remains the reference view: the STATUS column should read ONLINE and Fully operational for every member. An unexpected OFFLINE is the first signal to watch, tied to the threshold described in the roles and high availability guide.

Troubleshooting

Symptom	Likely cause	Fix
/1.0/metrics returns 403	Certificate not trusted	Add the certificate to the cluster trust store
Instance export fails for lack of space	Large archive on the workstation	Export to a mounted remote storage
No logs on the collector side	Incomplete syslog configuration	Fill in address, protocol and format
The node backup looks "empty"	You expected the instances	:backup saves the keys, not the instances; instances go through export

Glissez pour voir

Key points

• On Incus OS, backup and monitoring are driven remotely, through the API.
• Instances are backed up with snapshots (local) and exports (portable, offsite).
• The node backup (:backup) gathers the recovery keys: a critical secret for the vault, and worth testing by actually decrypting with it.
• Monitoring goes through an external Prometheus on /1.0/metrics, ideally with a metrics-only certificate.
• Logs are centralised to a remote syslog through the system API.
• system_is_ready and incus cluster list give the immediate health of the cluster.

Next steps

Roles, quorum and high availability Quorum, evacuation and cluster health.

A/B updates and rollback Update the cluster with no outage.

Shared CephFS storage Volumes that survive losing a node.

Incus OS without a shell The shell-less model, everything through the API.

×

📖 Voir la documentation →