Operations Center: deploy and drive an Incus OS fleet

Read this page in French

Managing one Incus OS node is done through its API. Managing a fleet calls for a central console. Operations Center is that console: an appliance that provisions servers, forms clusters, distributes updates and inventories instances, networks and storage, even in air-gapped environments. This guide deploys the appliance, then has it install and cluster three Incus OS nodes from start to finish, without ever building the cluster by hand. Intended audience: administrators building an Incus OS private cloud and looking for the sovereign equivalent of a virtualization console.

What you will learn

• Deploy the Operations Center appliance and avoid the trap that blocks its startup.
• Install and connect the management CLI.
• Generate a managed ISO whose nodes are born already enrolled.
• Let Operations Center form the cluster from ready servers.
• Drive the cluster (instances, networks) from the console.

Prerequisites

• The seed installation procedure for Incus OS, which this guide assumes you have already carried out once.
• A trusted client certificate for the API and the web interface.
• Enough capacity for four virtual machines (the appliance plus three nodes), 4 GB and 50 GB each for a lab.

Version used here

Everything below was carried out with Operations Center 0.7.3. The project releases often and publishes no detailed release notes, so behaviour can differ on a newer build. Check against your own version before relying on any specific detail.

Where Operations Center sits in the stack

Operations Center is a primary application of Incus OS, which makes it exclusive: a system runs either incus (a hypervisor) or operations-center (the console), never both. Operations Center lives on a dedicated appliance. Together with Migration Manager, it forms the management layer of the FuturFusion stack, designed as a free software alternative to closed platforms for people leaving VMware.

The guiding principle, worth absorbing before anything else: Operations Center deploys the nodes and forms the cluster, not the other way round. You never build the cluster by hand and hand it over afterwards, a point this guide returns to at the end.

Deploying the appliance

The appliance installs like an Incus OS node, with a seed that differs on two files. There is no incus seed, but an applications.json asking for operations-center, and an operations-center.json declaring at least one trusted client certificate. Without it, no authentication is possible later on.

applications.json

{"applications":[{"name":"operations-center"}]}

operations-center.json

{
  "version": "1",
  "apply_defaults": true,
  "trusted_client_certificates": ["-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----\n"]
}

With the usual install.json and network.json, you build the seed ISO and install the virtual machine. At startup, the application listens on port 8443 for both the API and the web interface.

Install from the current ISO

A system installed from an older ISO updates itself on first boot and reboots. For Operations Center, that reboot can land before initialization finishes and leave a half-created ZFS dataset (local/operations-center: dataset already exists): the application no longer starts, and with neither shell nor API there is no way to repair it. So install from the ISO of the current release: with no update on first boot, initialization runs in one pass and the API answers in under a minute.

Installing and connecting the CLI

Everything that follows is driven with the operations-center CLI, fetched from the project releases:

wget https://github.com/FuturFusion/operations-center/releases/download/v0.7.3/operations-center.linux.x86_64 \
  -O /usr/local/bin/operations-center && chmod +x /usr/local/bin/operations-center

You declare the appliance as a remote, then point the CLI at the trusted certificate (the one from the seed), without which authentication is refused:

operations-center remote add opscenter https://192.168.10.135:8443 --accept-certificate
cp client.crt ~/.config/operations-center/client.crt
cp client.key ~/.config/operations-center/client.key
operations-center remote switch opscenter
operations-center provisioning server list

At this stage the list contains only the appliance itself, self-registered as a server, in state ready.

Generating the managed ISO

This is the heart of the model. You create a provisioning token authorizing a number of installations, then ask Operations Center for a pre-seeded ISO built from that token:

operations-center provisioning token add --description "cluster lab" --uses 20
operations-center provisioning token list

operations-center provisioning token get-image <uuid> ./IncusOS-managed.iso \
  --application incus --architecture x86_64 --type iso

Operations Center assembles an ISO (around 3 GB) from its cached updates, embedding the incus application and a seed that automatically attaches the node to the appliance. A node installed from that ISO is born managed, with its own certificate covering its address: that detail is what makes everything afterwards run smoothly.

Installing the nodes and watching them turn ready

For each node of the future cluster, you install the virtual machine from the managed ISO, with no separate seed since everything is embedded. After first boot, each node registers itself with Operations Center. You watch them arrive:

operations-center provisioning server list

| Name                                 | Connection URL              | Type  | Status |
| 5919fc58-f510-42dd-93d3-a1636d80ae01 | https://192.168.10.131:8443 | incus | ready  |
| 842513e1-4d46-4294-b063-acebd05d36ab | https://192.168.10.168:8443 | incus | ready  |
| 0be9d0be-bafd-41e5-9fc2-24a0c5d38b46 | https://192.168.10.184:8443 | incus | ready  |
| operations-center                    | https://192.168.10.135:8443 | operations-center | ready |

The nodes arrive ready straight away, because Operations Center provisioned them itself and knows their certificate. You can then give them readable names:

operations-center provisioning server rename 5919fc58-... IncusOS01
operations-center provisioning server rename 842513e1-... IncusOS02
operations-center provisioning server rename 0be9d0be-... IncusOS03

Forming the cluster

The servers are ready but independent. You ask Operations Center to gather them into a cluster. The command takes an application configuration file carrying the client certificate and the listen address:

app-config.yaml

certificates:
  - type: client
    name: incus-admin
    certificate: |-
      -----BEGIN CERTIFICATE-----
      ...
      -----END CERTIFICATE-----
config:
  core.https_address: ":8443"

operations-center provisioning cluster add lab-incusos https://192.168.10.131:8443 \
  --server-names IncusOS01 --server-names IncusOS02 --server-names IncusOS03 \
  --application-seed-config app-config.yaml

The URL is that of the first server, and --server-names is repeated for each node. Operations Center then configures clustering across the three servers. You check on the console side:

operations-center provisioning cluster list

| Name        | Connection URL              | Status |
| lab-incusos | https://192.168.10.131:8443 | ready  |

And on the Incus side, the cluster is a genuine highly available cluster of three voters:

incus remote add lab https://192.168.10.131:8443 --auth-type tls --accept-certificate
incus cluster list lab:

| IncusOS01 | database-leader | ONLINE | Fully operational |
| IncusOS02 | database        | ONLINE | Fully operational |
| IncusOS03 | database        | ONLINE | Fully operational |

Driving the fleet from the console

Once the cluster is formed, everything that happens on it flows back into the Operations Center inventory. Launch an instance on the cluster and it shows up on the console side with its cluster and its host node:

incus launch images:debian/12 lab:demo01 --target IncusOS02
operations-center inventory instance list

| Name   | Project | Cluster     | Server    |
| demo01 | default | lab-incusos | IncusOS02 |

The inventory also covers networks (inventory network), ACLs, load balancers, DNS zones and storage: it is the single point from which you observe and govern the whole fleet, which is what the web interface under /ui/ reflects.

Do not graft Operations Center onto an existing cluster

It is tempting to build a cluster by hand and attach Operations Center afterwards. On the version used here (0.7.3), this does not work, and knowing it saves a lot of time.

Cluster members present a shared cluster certificate whose SAN entries do not cover their IP addresses. When Operations Center tests the return path to a node, TLS verification fails (certificate is valid for 127.0.0.1, not <ip>) and the server stays stuck in pending (registering). Regenerating the certificate to add the IPs then breaks trust (certificate signed by unknown authority), and there is no way back: deregistration is not supported and a node's registration state is frozen. The one reliable route is the one in this guide: let Operations Center provision the nodes from the start.

Reaching the interface and diagnosing

The web interface is served at https://<appliance>:8443/ui/, authenticated by the same client certificate. Import it into the browser in PKCS#12 format and make sure you present the trusted one: a wrong certificate yields a blank page (the UI loads, but every API call is rejected).

Since the appliance remains an Incus OS system, its system API answers on the same port under /os, which gives you the logs when nothing else responds:

curl -sk --cert client.crt --key client.key https://192.168.10.135:8443/os/1.0/debug/log

Filtering on the operations-center unit surfaces the application's real errors, even behind an empty interface. That is how both the dataset lockup and the certificate rejections get diagnosed.

Troubleshooting

Symptom	Likely cause	Fix
The appliance does not answer on 8443	ZFS dataset stuck after a first-boot update	Reinstall from the ISO of the current release
The CLI is refused (authentication mismatch)	CLI certificate not trusted	Point the CLI at the trusted certificate from the seed
A node stays pending (registering)	Cluster built by hand, certificate without IPs	Start over console-first: provision through the managed ISO
cluster add fails	app-config.yaml missing a certificate or an address	Provide both certificates and core.https_address

Glissez pour voir

Key points

• Operations Center is a primary application: a dedicated appliance, never a node.
• The model is console-first: Operations Center provisions the nodes, then forms the cluster.
• The managed ISO (token get-image) produces nodes that are born enrolled, in ready.
• Cluster formation goes through cluster add with an app-config.yaml.
• The whole cluster flows back into the inventory (instances, networks, storage).
• Grafting the console onto an existing cluster does not work on 0.7.3 (certificates, no deregistration).

Next steps

Multi-tenant private cloud Tenants, quotas and credentials on the cluster you just built.

OVN: VPCs and load balancers The virtual networks the inventory reports on.

Terraform: a three-tier stack Describe what runs on the fleet as code.

×

📖 Voir la documentation →