Incus OS: the immutable OS dedicated to Incus

Installing Incus on Debian means adding an application to a general-purpose system. Incus OS turns that around: it is an immutable system designed to run only Incus, with no shell, driven solely by an authenticated REST API. This page explains what Incus OS is and what it guarantees, then walks through what was actually done on a Proxmox lab: installation from ISO and seed, access to the built-in web interface, and above all building a three-node cluster entirely without the graphical interface. For administrators comfortable with Incus and virtualisation.

What you will learn

What Incus OS is, and why an immutable, shell-less system.
Install a node from an ISO image and seed files.
Reach the built-in web interface on port 8443.
Establish CLI client trust without going through the interface.
Build a three-node cluster entirely through seed and API.

What Incus OS is

The project describes Incus OS as an immutable OS designed solely to run Incus safely and reliably. It is not a distribution you install Incus onto: it is a system image where Incus and the kernel are integrated and signed, identical bit for bit across every machine.

It is built on Debian 13 using mkosi, with the project's own Incus and kernel builds. The break in philosophy fits in one sentence: no local or remote shell, administration going exclusively through an authenticated REST API. The lab described here runs Incus 7.2 (image IncusOS_202607010319).

Why a dedicated OS

A classic Incus server drifts over time: packages installed by hand, local tweaks, versions diverging from one machine to the next. Incus OS removes that deployment variance by shipping exactly the same software everywhere. Across a fleet of nodes, that is the guarantee of reproducible behaviour.

Immutability also brings a reduced attack surface: system partitions are read only and signed, and the absence of a shell closes a classic intrusion vector. The deliberate downside is that no third-party agent installs by hand, and no local adjustment is possible.

The key properties

Incus OS stacks several security and operational mechanisms that together define where it sits. The table sums them up before moving to practice.

Property	What it brings
UEFI Secure Boot and TPM 2.0	A verified and measured boot chain
Full disk encryption	LUKS backed by the TPM, plus ZFS encryption
Read-only, signed system partitions	The system cannot be altered in place
Atomic A/B updates	Switch to a new image, easy revert
REST API only, no shell	Authenticated administration, no interactive access

The A/B update model

This is one of the most distinctive points. Incus OS systems check for updates every 6 hours by default and apply them automatically. Two channels exist: stable, the default, which typically sees at least one weekly update picking up the latest stable Linux kernel bug-fix release and any relevant security issues, and testing, rebuilt roughly once a day.

The two halves of an update behave differently, and the distinction matters for operations: Incus itself updates automatically with a very short API interruption and no impact on running instances, while an OS update is staged and applies on the next reboot. The frequency is configurable, automatic updates can be disabled entirely, and scheduled downtime windows can be declared for applying application updates.

The A/B scheme writes the new version to a second partition: if something goes wrong, the system falls back to the previous one with no reinstall. It is the same principle as immutable container or mobile operating systems. The A/B updates and rollback guide covers reading versions, setting the channel and updating a cluster without an outage through the API.

The web interface is built in

Where Incus on Debian requires apt install incus-ui-canonical, Incus OS already ships the interface in its image. You open https://<IP>:8443/ui/ in a browser and the graphical management of instances, networks and pools is there, with nothing to install, which resolves the apparent contradiction with immutability.

That interface manages Incus, not yet the OS itself (reboots, system updates), which stay driven through the incus admin os subcommand. Above all, nothing forces you to use it: everything the interface does, the API and the remote CLI client do as well. The rest of this page demonstrates a complete deployment without ever opening the interface.

Installing a node from ISO and seed

Installation starts from the project's ISO image, accompanied by a set of seed files: small JSON files, burned onto a second medium labelled SEED_DATA, that drive the installation and the first configuration. The lab uses Proxmox virtual machines, but the seed logic is independent of the hypervisor.

Three hardware constraints were verified and are not negotiable:

The machine must be UEFI (OVMF) with Secure Boot available and a TPM 2.0.
The target disk must be virtio-scsi. A virtio-blk disk fails with no potential install devices found.
The SEED_DATA medium is mandatory: without it, the installer stops on unable to begin install from read-only device without seed configuration.

The seed files

A node needs three files. The first, install.json, triggers the installation:

{"force_reboot": false}

The second, network.json, pins a static IP address, which a cluster requires since member addresses have to be known in advance. It carries the network configuration flat, and the interface is identified by its MAC address:

{
  "version": "1",
  "interfaces": [
    {
      "name": "ens18",
      "hwaddr": "bc:24:11:98:92:a0",
      "addresses": ["192.168.10.131/24"],
      "routes": [{"to": "0.0.0.0/0", "via": "192.168.10.1"}],
      "roles": ["management"]
    }
  ],
  "dns": {"domain": "lab", "hostname": "node1", "nameservers": ["192.168.10.1", "1.1.1.1"]}
}

The third, incus.json, configures Incus. For a simple first node, apply_defaults is enough to create the local storage pool and the default network:

{"version": "1", "apply_defaults": true}

Those three files go into an ISO image labelled SEED_DATA, with genisoimage:

genisoimage -V SEED_DATA -J -r -o seed-node1.iso seed-node1/

The SEED_DATA volume name is not decorative: that is exactly the label Incus OS looks for to read the configuration.

How the installation unfolds

The installer writes the system to disk, then waits for the media to be removed: it does not power the machine off.

Boot from the ISO with the SEED_DATA medium attached as a second drive. The installer copies the system onto the virtio-scsi disk.
Wait for the message IncusOS was successfully installed. Please remove the install media. The machine stays on that screen.
Remove both media (install ISO and seed), then reboot from disk. Seed files other than install.json have been copied into the system; they apply on first boot.

On first boot, the node applies the static network and the Incus configuration. The API then answers on https://192.168.10.131:8443/1.0, with auth: untrusted as long as no client is trusted. The SSH port is closed: the concrete confirmation that there is no shell.

Establishing trust without the interface

On a node with no shell, a remote CLI client has to become trusted to drive Incus. The web interface offers an enrolment flow, but there is a route entirely through the seed: injecting a client certificate straight into the Incus configuration at first boot.

You first generate a client certificate and key pair, once, on the administration workstation:

openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:secp384r1 -nodes \
  -keyout client.key -out client.crt -days 3650 -subj "/CN=incus-admin"

Then you enrich the first node's incus.json with the preseed.certificates section, which takes the client certificate in PEM format:

{
  "version": "1",
  "apply_defaults": true,
  "preseed": {
    "certificates": [
      {"name": "incus-admin", "type": "client", "certificate": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----\n"}
    ]
  }
}

After reinstalling with that seed, the certificate is already trusted. A request carrying it confirms as much:

curl -sk --cert client.crt --key client.key https://192.168.10.131:8443/1.0 | grep -o '"auth":"[a-z]*"'
# "auth":"trusted"

No click in the interface, no image customisation tool: trust comes from the seed. That is the building block everything else rests on.

Building a three-node cluster without the interface

An Incus cluster gathers several servers under a single API, with a replicated configuration database and fault tolerance from three nodes onwards. Quorum and recovery are covered in the Incus cluster guide; this section focuses on what is specific to Incus OS: doing all of it without a shell and without an interface.

The administration workstation uses the incus client (here incus-client 6.0.4), configured with the already trusted certificate:

incus remote add node1 https://192.168.10.131:8443 --auth-type tls --accept-certificate

Bootstrapping the first node

The first node becomes a cluster of one. You give it a concrete cluster address, then enable clustering from the trusted client:

incus config set node1: cluster.https_address 192.168.10.131:8443
incus cluster enable node1: node1

Enabling the cluster regenerates the node's server certificate. The remote then points at a stale certificate, so you simply re-pin it:

incus remote remove node1
incus remote add node1 https://192.168.10.131:8443 --auth-type tls --accept-certificate

Issuing a token for each node

Every joining node needs a unique join token, issued by the first node. That token cannot be pre-generated in a static seed: it is the one step that goes through an API call, made from the trusted workstation, never from a shell on the nodes.

incus cluster add node1: node2

The command returns a base64-encoded token carrying the member name, the fingerprint of the cluster certificate and the contact addresses.

Having the nodes join through their seed

Nodes 2 and 3 join the cluster through their seed, without apply_defaults. The incus.json of a joining node describes the join. Two fields are indispensable on top of the token, and their absence was diagnosed directly in the Incus source:

cluster_certificate: the cluster certificate, in PEM format. The interactive client resolves it from the token on its own, but Incus OS applies the preseed without that resolution. Without the field, the join fails on No target cluster member certificate provided.
member_config with source: local/incus: the source of the zfs pool local on the system disk. Without it, Incus attempts a loop-backed pool, which Incus OS forbids (Loop backed pools aren't supported on IncusOS).

You fetch the cluster certificate presented by the first node:

openssl s_client -connect 192.168.10.131:8443 -showcerts </dev/null 2>/dev/null \
  | openssl x509 -outform PEM > node1-cluster.crt

Node 2's incus.json then looks like this, node 3 being identical with its own name and address:

{
  "version": "1",
  "apply_defaults": false,
  "preseed": {
    "cluster": {
      "enabled": true,
      "server_name": "node2",
      "server_address": "192.168.10.132:8443",
      "cluster_address": "192.168.10.131:8443",
      "cluster_token": "<token issued by node1>",
      "cluster_certificate": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----\n",
      "member_config": [
        {"entity": "storage-pool", "name": "local", "key": "source", "value": "local/incus"}
      ]
    }
  }
}

You install each node with its own seed, exactly like the first. On first boot, the node joins the cluster by itself. Since client certificate trust is replicated in the cluster database, the administration certificate is automatically recognised on the joined nodes: nothing to re-inject.

Checking the cluster

Once the three nodes are installed, the trusted client lists an operational cluster:

incus cluster list node1:

+-------+-----------------------------+------------------+--------+-------------------+
| NAME  |             URL             |      ROLES       | STATUS |      MESSAGE      |
+-------+-----------------------------+------------------+--------+-------------------+
| node1 | https://192.168.10.131:8443 | database-leader  | ONLINE | Fully operational |
| node2 | https://192.168.10.132:8443 | database         | ONLINE | Fully operational |
| node3 | https://192.168.10.133:8443 | database         | ONLINE | Fully operational |
+-------+-----------------------------+------------------+--------+-------------------+

All three members carry the database role: the three-node quorum is reached, and one node can fail without stopping the service. The local pool is a zfs pool sourced from local/incus on each member, and the SSH port stays closed everywhere. The whole cluster was built without ever opening the web interface.

Traps encountered

These errors come from the real lab. Knowing them saves a reinstall cycle.

Symptom	Cause	Fix
`no potential install devices found`	Target disk is virtio-blk	Use a virtio-scsi disk
`unable to begin install from read-only device without seed configuration`	`SEED_DATA` medium missing	Burn the seed ISO with the `SEED_DATA` label
`No target cluster member certificate provided`	`cluster_certificate` missing from the join seed	Add the cluster certificate PEM to the preseed
`Loop backed pools aren't supported on IncusOS`	No `member_config` for the `local` pool	Supply `source: local/incus` as `member_config`

How it differs from Incus on Debian

The choice is not "better or worse" but two distinct operating models. The table helps decide.

	Incus on Debian	Incus OS
Base	A Debian you manage	An immutable image, provided
System access	Full shell (SSH, sudo)	REST API only
Updates	`apt`, at your own pace	Atomic A/B, automatic
Host customisation	Total	Deliberately none
Reproducibility	Your responsibility	Guaranteed, bit for bit

When to choose it

Incus OS targets deployments where reproducibility and security outrank flexibility: a fleet of identical nodes, an infrastructure you want immutable and self-updating, a context where the absence of a shell is a compliance asset. For a single server you like to tinker with, or to discover Incus, the classic installation on Debian remains the better fit.

Key points

Incus OS is an immutable system dedicated to Incus, built on Debian 13, with signed Incus and kernel builds.
No shell: administration through an authenticated REST API only, SSH port closed.
Installation from ISO and seed; a virtio-scsi disk and a SEED_DATA medium are both mandatory.
Client trust is injected through the seed (preseed.certificates), without the interface.
A three-node cluster is built with seed and API: joining needs cluster_certificate and member_config with source: local/incus.

Next steps

A/B updates and rollback: drive the atomic switch of a whole cluster through the API, and step back if it disappoints.
Roles, quorum and high availability: understand voters and stand-by members so a lost node does not take the distributed database down.
Workers and remote nodes: add compute capacity without weighing down the quorum, with targeted placement and failure domains.
Backup and monitoring: recovery keys, instance exports and Prometheus metrics, the only safety nets without SSH access.