Shared CephFS storage for an Incus OS cluster

In an Incus cluster, an instance can be restarted or moved onto any node. If its data lives on the local disk of one node, it does not follow. The answer is shared storage: a volume reachable from every node at once. This guide shows how to attach a CephFS (the distributed filesystem of Ceph) to the instances of an Incus OS cluster, entirely through the API, without ever opening a shell on the nodes. Intended audience: administrators of an Incus OS cluster who already run a Ceph cluster.

What you will learn

Why shared storage becomes necessary as soon as you have a cluster.
Install the Ceph client on the nodes with the incus-ceph add-on.
Declare a Ceph cluster in Incus OS through the ceph service (API).
Attach a CephFS to an instance with a disk device.
Fix write permissions for an unprivileged container.

Prerequisites

A working Incus OS cluster. See Incus OS without a shell and Incus cluster.
A Ceph cluster reachable from the nodes, with a CephFS already created (called labfs here) and a dedicated user (here client.incus). The Ceph documentation covers building the cluster, the pools and the filesystem.
An Incus client whose certificate is trusted, with a remote to each node.

As with A/B updates, all Incus OS administration goes through its REST API prefixed with /os, queried with incus query. The Ceph service is configured per node: the commands below target each cluster member through its remote.

Why shared storage in a cluster

A local pool (ZFS on a node's system disk) is perfect for an instance that stays on its node. But the value of a cluster is resilience: if a node fails, you want to restart its instances elsewhere. Without shared storage, the data volumes stay trapped on the failed node.

CephFS answers that need. It is a filesystem mounted simultaneously by several clients in read-write. Two instances, on two different nodes, see and modify the same files. If you know Kubernetes, it is the equivalent of an RWX (ReadWriteMany) volume served by a distributed storage class. Ceph also offers block storage (RBD, closer to ReadWriteOnce) through source=ceph:, but for a volume shared between instances, CephFS is what you want.

Installing the Ceph client on the nodes

Incus OS does not embed the Ceph client by default. You install the incus-ceph application, an add-on that adds the Ceph binaries to the immutable system. You trigger it with a POST on the applications endpoint, for each node:

for n in node1 node2 node3; do
  incus query -X POST "$n:/os/1.0/applications" -d '{"name":"incus-ceph"}'
done

The response is empty on success. Then check the application is present on every node:

incus query node1:/os/1.0/applications

The list should hold /os/1.0/applications/incus-ceph next to /os/1.0/applications/incus. Until that add-on is installed, the ceph service described below stays unavailable.

Declaring the Ceph cluster

Once the client is present, you declare the Ceph cluster through the ceph service. You supply three things: the FSID of the Ceph cluster, the address of at least one monitor, and the key of the Ceph user. Those values come from the Ceph side (ceph fsid, ceph mon dump, and ceph fs authorize labfs client.incus / rw for the key).

The configuration is applied with a PUT on services/ceph, on each node:

cat > /tmp/ceph.json <<'JSON'
{
  "config": {
    "enabled": true,
    "clusters": {
      "ceph": {
        "fsid": "f2a9d236-38b5-4b79-a111-da7e41163404",
        "monitors": ["192.168.10.202"],
        "keyrings": {
          "incus": { "key": "<the key returned by ceph fs authorize>" }
        }
      }
    }
  }
}
JSON

for n in node1 node2 node3; do
  incus query -X PUT "$n:/os/1.0/services/ceph" -d "$(cat /tmp/ceph.json)"
done

From that configuration, Incus OS writes /etc/ceph/ceph.conf itself (with the FSID and the monitor) and the client keyring file. The cluster name, here ceph, is the key of the clusters object; that is what you reuse as ceph.cluster_name when attaching the volume.

Attaching the CephFS to an instance

Mounting is done with a disk device whose source starts with cephfs:. You give the filesystem and a subpath, the Ceph user and the cluster name declared above:

incus config device add node1:cephtest cephvol disk \
  source=cephfs:labfs/incusdata \
  ceph.user_name=incus \
  ceph.cluster_name=ceph \
  path=/data

The instance then mounts the CephFS on /data. A df from inside confirms it:

incus exec node1:cephtest -- df -hT /data

Filesystem                                         Type  Size  Used Avail Use% Mounted on
incus@f2a9d236-38b5-4b79-a111-da7e41163404.labfs=/ ceph   19G     0   19G   0% /data

The ceph type and the FSID in the mount name confirm the CephFS kernel client is active and that cephx authentication succeeded.

Fixing write permissions

The mount succeeds, but an unprivileged container often fails to write, with a Permission denied. The cause is the Incus UID mapping: the container's root (uid 0) is projected onto the host, and therefore onto CephFS, as uid 1000000 by default. If the CephFS root belongs to root, that shifted uid has no rights.

The clean solution is to prepare a dedicated subdirectory owned by the mapped uid, from a client that already mounts the CephFS as administrator:

# On a host with the Ceph client, mounted on /mnt/cephfs:
mkdir /mnt/cephfs/incusdata
chown 1000000:1000000 /mnt/cephfs/incusdata

You then point the disk device at that subpath (source=cephfs:labfs/incusdata, as above). The container writes without error, and the file shows up on the Ceph side owned by 1000000:

incus exec node1:cephtest -- sh -c 'echo hello > /data/hello.txt'

The point of CephFS shows when several clients reach the same path. A file dropped on the Ceph side, or by another instance, is immediately visible inside the instance:

incus exec node1:cephtest -- ls -l /data/

A file created by the instance appears the same way on the Ceph side and to any other client mounting labfs. That shared consistency is what lets an instance restarted on another node find its data intact, provided the same disk device is attached in its profile or its configuration.

Troubleshooting

Symptom	Likely cause	Fix
`errno 95` or `no keyring found` at mount time	`keyrings` key named `client.incus` instead of `incus`	Correct the ceph service `PUT`; the map key is the username alone
The `ceph` service returns `not found`	`incus-ceph` add-on missing on the node	Install the application with `POST /os/1.0/applications`
`Permission denied` when writing from a container	`root` UID mapped to `1000000`	Create a subdirectory owned by `1000000` and point the device at it
`error connecting to the cluster`	Monitor unreachable from the node	Check the network route and the address in `monitors`

Key points

A cluster needs shared storage so instances keep their data when they change node.
CephFS provides a shared volume (RWX); RBD (source=ceph:) provides block storage (RWO).
On Incus OS, everything goes through the API: the incus-ceph add-on, then the ceph service with FSID, monitors and keyring.
The keyrings object key is the username alone (incus), not client.incus.
You attach the volume with a disk device using source=cephfs:<fs>/<path>.
A container writes with a mapped UID (1000000): align the ownership of the subdirectory.

Next steps

Incus cluster Quorum, token joins and recovery after an outage.

Roles, quorum and HA Voters, stand-by members and auto-healing.

Incus OS without a shell Installation, seed and clustering with no shell.

Incus storage Local dir, ZFS and LVM pools, and volumes.