Incus OS cluster: roles, quorum and high availability

Read this page in French

An Incus OS cluster spreads its state across a distributed database. To stay available despite a node failure, that database elects a quorum and assigns roles to its members. This guide explains how to read those roles, how Incus keeps the quorum, how to tune high availability (the number of voters and stand-by members), and how to evacuate then restore a node for maintenance without an outage. Intended audience: administrators running an Incus OS cluster of three nodes or more.

What you will learn

• How Incus replicates its state and why the quorum matters.
• The member roles: voter, stand-by, client, leader.
• Read your cluster roles and tune them (max_voters, max_standby).
• Fault tolerance: offline_threshold and auto-healing.
• Evacuate and restore a member for maintenance.

Prerequisites

• A three-node Incus OS cluster. See Incus OS without a shell for building it and Incus cluster for the general concepts.
• A trusted Incus client with a remote to one cluster member.

Every command below runs from the administration workstation, through a remote pointing at a member (here node1:). The cluster answers consistently whichever member you query.

How Incus replicates its state

Incus stores its configuration, its instances and its networks in a replicated database (Cowsql, derived from SQLite) synchronised by the Raft algorithm. Every change has to be confirmed by a majority of the voting members, which is what the quorum means. As long as that majority answers, the cluster accepts writes; if it is lost, the cluster falls back to read only to protect consistency.

That is why a cluster is built with at least three members. With three voting members, the cluster survives losing one of them: two votes out of three still form a majority. With only two members, a single failure loses the quorum, hence the constant recommendation of three nodes.

The member roles

Not all members relate to the database the same way. Incus assigns roles that decide who votes, who replicates and who simply consumes:

Role	Replicates the database	Votes (quorum)	Can become leader
database (voter)	yes	yes	yes
database-leader	yes	yes	it is the current leader
database-standby	yes	no	no, but promoted to voter if a voter fails
database-client	no	no	no

Glissez pour voir

The leader is the voter that coordinates writes and watches the health of the others. Stand-by members replicate the database quietly, ready to be promoted to voter if one disappears. Client members do not carry the database at all: they access it remotely and mainly serve to host instances. That last role is the key to adding capacity without weighing down consensus, the subject of extending a cluster with remote nodes.

Reading your cluster roles

The incus cluster list command gives a synthetic view of members and their roles:

incus cluster list node1:

+-------+-----------------------------+-----------------+--------+-------------------+
| NAME  |             URL             |      ROLES      | STATUS |      MESSAGE      |
+-------+-----------------------------+-----------------+--------+-------------------+
| node1 | https://192.168.10.131:8443 | database        | ONLINE | Fully operational |
| node2 | https://192.168.10.132:8443 | database-leader | ONLINE | Fully operational |
|       |                             | database        |        |                   |
| node3 | https://192.168.10.133:8443 | database        | ONLINE | Fully operational |
+-------+-----------------------------+-----------------+--------+-------------------+

Here the three members vote (database), and node2 is the current leader. For detail on a single member, incus cluster show exposes its failure domain, its groups and whether it carries the database:

incus cluster show node1:node4

roles: []
failure_domain: default
groups:
- default
database: false
status: Online

A member whose roles is empty and whose database is false is a database-client: it belongs to the cluster and hosts instances, but stays outside consensus.

Tuning high availability

Two global settings drive how roles are spread: the number of voters (cluster.max_voters, 3 by default) and the number of stand-by members (cluster.max_standby, 2 by default). Incus assigns roles automatically within those limits, voter first, then stand-by, then client for the rest.

The effect is immediate and observable. On a cluster where a fourth member was a database-client because stand-by slots were disabled, restoring the default value flips it straight away:

incus config set node1: cluster.max_standby 2
incus cluster list node1: -f csv | grep node4

node4,https://192.168.10.134:8443,database-standby,x86_64,default,,ONLINE,Fully operational

The member moved from client to stand-by as soon as a slot opened. Conversely, setting cluster.max_standby to 0 forces members beyond the three voters to stay client. That lever decides whether an extra node strengthens database resilience (stand-by) or merely adds compute capacity (client).

Do not inflate the voter count blindly

Raising cluster.max_voters above three does not always improve availability: every extra voter has to replicate and vote on each write, which weighs down consensus. Three voters plus two stand-by members is the recommended default balance. Keep voters on nodes that are fast and close to one another.

Fault tolerance and auto-healing

A member that stops responding is marked offline after a delay set by cluster.offline_threshold, whose default is 20 seconds (minimum 10). That threshold has to stay consistent with network quality: a slow or saturated link can make a healthy member look failed.

By default, Incus simply marks the member offline. Setting cluster.healing_threshold to a non-zero value (it is 0, therefore disabled, by default) enables auto-healing: once the member is considered offline, its instances are automatically evacuated to other nodes. That is useful for a fleet where workloads should restart automatically, provided you have shared storage such as CephFS so the data follows.

This auto-healing remains an infrastructure mechanism, not a Kubernetes equivalent: Incus moves machines (virtual machines and system containers), it does not watch the health of an application process inside them and does not restart it. For application-level restarts, the application has to carry its own logic, for instance systemd with Restart=, or an application orchestrator installed on top.

The trickiest part of auto-healing is split brain. During a partial network partition, the leader can declare a node offline and evacuate its instances while that node is still running and keeps writing to shared storage. When the link comes back, two copies of the same instance have written to the same volumes, causing corruption (orphan blocks, destroyed inodes, detectable with e2fsck). Incus hardened this behaviour (the fix tracked in issue lxc/incus#1032, shipped in Incus 6.4), but the risk only disappears with hardware fencing.

Auto-healing on shared storage: plan for fencing

Only enable cluster.healing_threshold if you can guarantee that a node declared offline is genuinely stopped. The countermeasure is fencing: software that listens for the cluster-member-healed event and cuts power to the suspect node through its BMC (IPMI or Redfish) or a switchable PDU, before its ghost instances corrupt the storage. Without that guard rail, leave the threshold at 0 and evacuate by hand.

Evacuating and restoring a member

Before maintenance (hardware upgrade, replacement), you evacuate a member: Incus stops or migrates its instances to other nodes, then puts it in the EVACUATED state. The quorum is unaffected as long as a majority of voters stays online.

1. Evacuate the target member.

   incus cluster evacuate node1:node4

   The output details every instance moved, for example Migrating "wl-node4" to "node2".

2. Check the member is in maintenance and its instances have migrated.

   incus cluster list node1: -c ns
   incus list node1: -c ns,L

   The member shows as EVACUATED and its instances now run on other nodes.

3. Restore the member once maintenance is done.

   incus cluster restore node1:node4

   Incus puts it back ONLINE and brings home the instances that belonged to it.

That evacuate then restore cycle is the basic gesture of cluster operations: it lets you work on a node without a service outage, the workloads being carried temporarily by the other members.

Troubleshooting

Symptom	Likely cause	Fix
The cluster refuses writes	Quorum lost (too many voters offline)	Bring enough voters back online; as a last resort, incus cluster recover
A healthy member goes OFFLINE	offline_threshold too low for the network	Raise cluster.offline_threshold
A new member stays database-client	cluster.max_standby at 0, or slots full	Raise max_standby if you want it to replicate
Instances do not migrate on evacuation	Local, unshared storage	Use shared storage (CephFS) for mobility

Glissez pour voir

Key points

• Incus replicates its state with Raft; writing needs a majority of voters (the quorum).
• Three voters survive the loss of one member; that is the recommended baseline.
• The roles: voter (votes), stand-by (replicates, promoted when needed), client (hosts without voting), leader (coordinates).
• cluster.max_voters (3) and cluster.max_standby (2) decide the spread, with immediate effect.
• cluster.offline_threshold (20 s) and cluster.healing_threshold (0, disabled) govern detection and auto-healing.
• Auto-healing acts on machines, not processes: it is not the application-level self-healing of Kubernetes.
• Only enable auto-healing with fencing (BMC or PDU) to avoid split brain and corruption of shared storage.
• Evacuate then restore a member to perform maintenance without an outage.

Next steps

Workers and remote nodes Add database-client nodes for compute capacity.

Incus OS without a shell Build the base cluster, with no shell.

Shared CephFS storage Make instances mobile between nodes.

A/B updates and rollback Update the cluster with no outage.

×

📖 Voir la documentation →