Incus cluster: multi-node high availability

Read this page in French

An Incus cluster gathers several servers under a single API: instances spread across the nodes, the configuration is replicated in a distributed database, and a node can go down without stopping the service. This guide builds a cluster step by step: exposing the API, enabling clustering, adding nodes with a token, and above all understanding the three-node quorum and how to recover after an outage. Built and tested on a three-node Incus 7.0 cluster (--target placement, instance migration, evacuate and restore). For people aiming at high availability.

What you will learn

• Expose the API and enable clustering.
• Add a node through a join token.
• Understand the quorum, and why three nodes.
• Maintain and recover a cluster.

Prerequisites

• Freshly installed Incus servers, on identical versions (a version gap makes the join fail).
• Port 8443 open between the nodes.
• Synchronised clocks (NTP): a drift breaks replication.

Exposing the API and enabling clustering

By default, Incus listens only on its local socket. Clustering requires the API to be exposed on the network through core.https_address, otherwise enabling it fails with This server is not available on the network:

incus config set core.https_address=:8443

The recommended route for a fresh cluster is incus admin init, answering yes to "use Incus clustering". On a server that is already initialised, you turn the daemon into a single-node cluster:

incus cluster enable node1
incus cluster list

+-------+-----------------------------+-----------------+--------+-------------------+
| NAME  |             URL             |      ROLES      | STATUS |      MESSAGE      |
+-------+-----------------------------+-----------------+--------+-------------------+
| node1 | https://192.168.10.184:8443 | database-leader | ONLINE | Fully operational |
|       |                             | database        |        |                   |
+-------+-----------------------------+-----------------+--------+-------------------+

The first member carries the database-leader and database roles: for now it hosts the distributed database on its own.

Adding a node

Adding happens in two steps, with a single-use token.

1. On a cluster member, generate the token for the future node:

   incus cluster add node2
   # Member node2 join token: eyJzZXJ2ZXJf...

   The token carries the member addresses, a secret and the fingerprint of the cluster certificate.

2. On the new node, run incus admin init, answer yes to "joining an existing cluster" and paste the token. For automation (Ansible, cloud-init), a preseed performs the join without interaction. You must expose the API of the joining node first, otherwise the join fails, as the warning below explains:

   # on the joining node, BEFORE the preseed
   incus config set core.https_address=192.168.10.50:8443

   # preseed.yaml on node2 (192.168.10.50)
   cluster:
     enabled: true
     server_address: 192.168.10.50:8443
     cluster_token: eyJzZXJ2ZXJf...
     member_config:
       - entity: storage-pool
         name: default
         key: source
         value: ""

   cat preseed.yaml | incus admin init --preseed

The preseed that succeeds without doing anything (exit 0, no join)

Three traps can block a real three-node build, and all three are silent:

• core.https_address must be set on the joining node before the preseed. Without it, the join stops on Cannot use wildcard core.https_address "" for cluster.https_address. The preseed's server_address does not set it for you.
• A node already initialised standalone cannot join. If the server has already run incus admin init --auto, the preseed returns exit 0 doing nothing, and the member never appears in incus cluster list. It has to be wiped: systemctl stop incus.service incus.socket, rm -rf /var/lib/incus, restart, then replay the join.
• The token prints on the line after its label. incus cluster add node2 prints Member node2 join token: and the token on the next line. A script parsing the label line gets an empty string, and the join fails on ServerName is required. Read the line starting with eyJ.

Another common trap is an expired token. Too long a delay between incus cluster add and the join, or a server restart in between, invalidates it (error No matching cluster join operation found). Consume the token immediately and regenerate it when needed; its lifetime is set by cluster.join_token_expiry, which defaults to 3 hours.

The member_config supplies the values that are specific to the node, such as the storage pool source: each server keeps its own local disks while sharing the configuration.

The quorum: why three nodes

This is the point to grasp before aiming at high availability. The Incus distributed database (Cowsql, a fork of dqlite) is replicated by the Raft algorithm, which needs a majority of voters to work.

• By default, 3 members are voters (cluster.max_voters=3).
• The database stays available as long as a majority is online. With 3 voters, the cluster survives losing one node.
• With only 2 nodes, there is no real fault tolerance (losing one voter loses the majority).

The conclusion is blunt: a real HA cluster starts at 3 nodes. Below that, you get management convenience, not high availability.

Once node2 and node3 have joined, incus cluster list shows the three members ONLINE. The database-leader role stays on node1, the other two carry database: together they form the three-vote Raft quorum.

+-------+-----------------------------+-----------------+--------+-------------------+
| NAME  |             URL             |      ROLES      | STATUS |      MESSAGE      |
+-------+-----------------------------+-----------------+--------+-------------------+
| node1 | https://192.168.10.184:8443 | database-leader | ONLINE | Fully operational |
|       |                             | database        |        |                   |
+-------+-----------------------------+-----------------+--------+-------------------+
| node2 | https://192.168.10.50:8443  | database        | ONLINE | Fully operational |
+-------+-----------------------------+-----------------+--------+-------------------+
| node3 | https://192.168.10.199:8443 | database        | ONLINE | Fully operational |
+-------+-----------------------------+-----------------+--------+-------------------+

Placing instances

Left alone, Incus places a new instance on the member carrying the fewest. You target a specific node with --target:

incus launch images:debian/13 web1 --target node2
incus launch images:debian/13 web2 --target node3
incus launch images:debian/13 web3            # automatic placement
incus list -c ns4L   # the LOCATION column shows the running node

On the test cluster, the first two instances land where they are sent, and the third is placed automatically on the least loaded node, here node1:

+------+---------+----------------------+----------+
| NAME |  STATE  |         IPV4         | LOCATION |
+------+---------+----------------------+----------+
| web1 | RUNNING | 10.107.24.164 (eth0) | node2    |
| web2 | RUNNING | 10.107.24.252 (eth0) | node3    |
| web3 | RUNNING | 10.107.24.100 (eth0) | node1    |
+------+---------+----------------------+----------+

Maintenance and recovery

To work on a node, you evacuate it, so its instances migrate or stop, then you restore it:

incus cluster evacuate node2
# ... maintenance ...
incus cluster restore node2

On the test cluster, evacuating node2 migrates web1 to node1: Incus stops the instance, moves it, then marks the node EVACUATED. The member stays in the cluster but accepts no more instances:

Evacuating cluster member: Stopping "web1" ... Migrating "web1" to "node1"

+-------+-----------------------------+-----------------+-----------+--------------------------------+
| NAME  |             URL             |      ROLES      |  STATUS   |            MESSAGE             |
+-------+-----------------------------+-----------------+-----------+--------------------------------+
| node2 | https://192.168.10.50:8443  | database        | EVACUATED | Unavailable due to maintenance |
+-------+-----------------------------+-----------------+-----------+--------------------------------+

The restore walks the same path backwards: web1 migrates back to node2, which returns to ONLINE. Evacuation behaviour is set per instance through cluster.evacuate (migrate live when storage allows it, stop, or force-stop).

To remove a member for good, add --force if it is unreachable:

incus cluster remove node2

Recovering after losing quorum

If you lose the majority, for instance 2 nodes out of 3, the cluster becomes unavailable. Recovery is possible as long as one database member survives:

1. incus admin cluster list-database, to spot the survivors;
2. systemctl stop incus.service incus.socket on every survivor;
3. on the chosen new leader: incus admin cluster recover-from-quorum-loss;
4. restart incus.socket then incus.service, leader first.

No data is removed from the database.

Key points

• Clustering first requires the API to be exposed: incus config set core.https_address=:8443.
• You enable it with incus admin init (recommended) or incus cluster enable (single node).
• A node is added through incus cluster add (token) then a join on the new server; the token must be consumed quickly.
• Real HA needs 3 nodes (Raft and Cowsql quorum); 2 nodes tolerate no failure at all.
• Identical versions and NTP are mandatory; recovery goes through recover-from-quorum-loss.

FAQ: common questions about Incus clusters

How many nodes does a highly available Incus cluster need?

Three nodes minimum. The Incus distributed database (Cowsql, a fork of dqlite) is replicated by the Raft algorithm, which needs a majority of voters. By default three members are voters, so the cluster survives losing one node.

How do you add a node to an Incus cluster?

In two steps. On an existing member, generate a single-use token with incus cluster add node-name. On the new server, run incus admin init answering yes to joining an existing cluster and paste the token, or feed a YAML preseed carrying the cluster section.

Why does my Incus cluster join fail with No matching cluster join operation?

Because the join token expired. That is the classic trap: too long a delay between incus cluster add and the join, or a server restart in between, invalidates the token. Consume it immediately and regenerate it if needed. Its lifetime is set by cluster.join_token_expiry, which defaults to 3 hours.

How do you recover an Incus cluster after losing quorum?

Recovery is possible as long as one database member survives. Identify the survivors with incus admin cluster list-database, stop incus.service and incus.socket on all of them, then run incus admin cluster recover-from-quorum-loss on the chosen leader. No data is removed from the database.

Next steps

• Incus OS cluster: roles and HA: the same quorum, on the shell-less immutable base.
• Shared CephFS storage: what makes instances mobile between nodes instead of pinned to one.
• OVN: VPCs and load balancers: the distributed networks a cluster makes possible.
• Multi-tenant private cloud: turning the cluster into a sovereign IaaS.

×

📖 Voir la documentation →