dsoxlab for the trainer: the infrastructure of vm labs

A catalogue ships no Terraform and no cloud-init: it declares its machines in the infra: block of its meta.yml, and dsoxlab does the rest with the templates packaged in the tool. This lesson is written for whoever brings up the machines that vm labs need, the trainer of a class or the author testing their catalogue: the topology, the SSH key per clone, the choice of the provider, KVM, Incus or Outscale, nested virtualisation, the snapshots and the machines that outlive their state. It describes version 0.2.5, and ends on a derived use: getting throwaway machines without writing a single exercise.

What you will learn

Declare a topology of machines in meta.yml, and what the contract imposes on the network name.
Generate the SSH key that provision requires, once per clone.
Choose a provider, and know that each keeps its own state.
Enable nested virtualisation in the right place when dsoxlab runs inside a VM.
Understand the two accounts created on every machine, the snapshots and the orphans.
Repurpose dsoxlab as a provisioner of throwaway VMs, with no lab.

Only vm labs need any of this

A catalogue made of shell labs needs no infrastructure: the exercise runs on the learner's machine, dsoxlab provision is never called, and the meta.yml carries no infra: block. That is a conforming catalogue, not an incomplete one, and the site's Terraform catalogue is the example. Everything that follows concerns catalogues that declare at least one lab in runtime.type: vm.

The infrastructure is packaged in the tool

The Terraform modules of the three providers, kvm, incus and outscale, and the cloud-init templates of the packaged distributions, AlmaLinux, Ubuntu and Debian, live inside dsoxlab. provision copies them to ~/.local/state/dsoxlab/<catalog-id>/, generates a variables file from the meta.yml, and runs Terraform there. The state never lands in the lab repository. A catalogue therefore declares this, and nothing more:

# meta.yml, infra block
infra:
  provider: kvm                 # or a list of candidates
  network: lab-linux            # libvirt network of this catalogue
  cidr: 10.10.10.0/24
  hosts:
    - name: alma-1.lab
      distro: alma10
      ram_mb: 2048
      vcpu: 2
      disk_gb: 20
      extra_disk_gb: 5          # second disk (/dev/vdb), for LVM or RAID labs

Do not declare IP addresses: they come from the Terraform outputs, and the inventory is derived from them. Every catalogue should own its own libvirt network, so that two catalogues never fight over the same subnet. The packaged distributions are alma10, alma9, ubuntu26, ubuntu24, ubuntu22, debian13 and debian12; that name drives the image and the cloud-init.

One key of infra.providers.kvm deserves to be known because a fresh machine needs it: storage_pool, the libvirt pool where the volumes are created, default by default. On a fresh Ubuntu 24.04, virsh pool-list --all is empty, and provision stops on a raw Pool Not Found. Two ways out: create the default pool, and dsoxlab doctor prints the commands that do it, or point that key to a pool you already own. The packaged template is never edited.

The SSH key is per clone, not per catalogue

provision deploys <catalogue>/ssh/id_ed25519.pub on every node, and refuses to start without the private half next to it. Both come from dsoxlab instructor bootstrap, which regenerates the pair whenever either half is missing, and checks in passing that Terraform and ansible-runner are installed. A public key committed by the author would serve nobody, its private half staying on their disk, and the published catalogues ignore the whole ssh/ directory.

dsoxlab instructor bootstrap    # generates <catalogue>/ssh/id_ed25519 if missing

Every machine that provisions a catalogue therefore plays that command once, learners included: the word "instructor" names the command, not its audience. It is the most frequent omission after a catalog add, and start makes it visible by naming the step that broke.

Supported versions

The floor is libvirt 8.0, and it was not chosen from a compatibility table: the three versions provisioned for real, and the machine had to answer over SSH, not merely see Terraform not complain. libvirt 8.0 in an Ubuntu 22.04 VM, 9.0 in a Debian 12 VM, 10.0 on the reference machine with a real vm lab of the Linux catalogue. Nothing below 8.0 has been tried, and that is the only reason a floor remains.

Component	Supported	How it was established
libvirt	8.0 or later	Real provisioning on 8.0, 9.0 and 10.0, with an SSH answer
Terraform provider `dmacvicar/libvirt`	`~> 0.9`	The constraint the packaged template declares; no known floor inside that range

dsoxlab doctor checks that floor and refuses a version below it, naming the cause. It also prints the provider version actually pinned for this catalogue, the one terraform init wrote in the state, and not the one the template asks for: two machines honouring ~> 0.9 can be running different versions, and that version appears in dsoxlab support. The template explicitly names the EFI firmware loader of its machines, which prevents an older libvirt from reading it back wrongly and imposing a higher provider floor.

Running dsoxlab inside a virtual machine

A vm lab needs /dev/kvm. Inside a virtual machine, that means nested virtualisation, and nesting is a property of the host, not of the guest: nothing installed in the guest produces it. It is enabled outside, guest powered off, and the place depends on the hypervisor that hosts dsoxlab.

Host	Where it is enabled
KVM, libvirt or Incus	`/sys/module/kvm_intel/parameters/nested`, or `kvm_amd`, must read `Y`; an `options kvm_intel nested=1` under `/etc/modprobe.d/` makes it permanent
VMware Workstation or Fusion	Virtualize Intel VT-x/EPT, in the VM's processor settings, machine powered off
VirtualBox	Nested VT-x/AMD-V, which depends on the CPU, and is unavailable on a Windows where Hyper-V or WSL2 already holds the hypervisor
macOS on Apple Silicon	Neither VirtualBox nor KVM exists there, and the packaged images are x86-64: it is a separate road, not a box to tick

dsoxlab doctor names that case rather than leaving you to guess. When /dev/kvm is missing, it first looks at where it is running, and inside a virtual machine it says that nesting is unavailable, naming the detected hypervisor; on a physical machine, it sends you to the BIOS. It used to offer both at once, which sent half its readers to visit a BIOS their machine does not have. For the sizing of the guest, 4 vCPU and 8 GB are measured, and the lesson Sizing your machine for the labs gives the detail.

Getting started, and choosing a provider

Six commands cover the whole life cycle, and doctor sorts its findings into two tables whose classification depends on three facts only, never on the domain: does the catalogue have vm labs, which provider is active, which providers does it declare. A hypervisor this catalogue does not use never shows up in red.

dsoxlab instructor bootstrap    # the key, once per clone
dsoxlab doctor                  # what this catalogue requires, and what is missing
dsoxlab provision               # terraform apply on the current provider
dsoxlab infra status            # is every declared host reachable, and if not why
dsoxlab ssh alma-1.lab          # an interactive session on one of them
dsoxlab destroy                 # tear everything down

provision --host <fqdn> targets a single machine, and the option is repeatable; without it, the whole plan is applied, the shared resources, network and base images, being handled by Terraform's dependency graph. The provider resolves by the first rule that matches: the DSOXLAB_PROVIDER variable, then the context set by dsoxlab use --provider, then a meta.yml that declares only one. Several candidates with no explicit choice is not an error: only the infrastructure commands refuse to proceed, and they say so.

dsoxlab use --provider kvm                 # durably, for this catalogue
DSOXLAB_PROVIDER=incus dsoxlab provision   # for one command

Each provider keeps its own Terraform state, under ~/.local/state/dsoxlab/<catalog-id>/terraform/<provider>/. Switching provider therefore does not destroy what the other one holds, which is convenient, and also how one forgets a running fleet. dsoxlab infra status is the habit that costs nothing.

Two accounts, and what it changes for the labs

cloud-init creates the same two accounts on every node, hardened the same way: member of wheel or sudo, sudo NOPASSWD:ALL, SSH key only, no login password, ssh_pwauth: false. The separation is deliberate, for traceability and revocation.

Account	Role
`ansible`	The service account of the automation. It is what dsoxlab and the labs' playbooks connect as, carried over into the generated `ssh_config`
`student`	The human account, on the machine the learner drives

The consequence for lab authors is concrete: anything that restricts login, an AllowUsers in sshd_config or a remote_user, must target ansible, never student, or the next dsoxlab command locks itself out.

Snapshots

snapshot_required: true in a lab's runtime commits the tool, it does not inform it. run takes a restore point of the disk before playing setup.yaml, and fails with code 2 if it cannot: a lab that asks for a safety net does not start without one. reset returns the machine to that point rather than replaying cleanup.yaml, then replays setup.yaml. clean removes the restore point, and with it the overlay file it had created.

On the kvm provider, it is an external disk snapshot, never an internal one, because the template boots its machines in UEFI and libvirt refuses internal snapshots on a pflash firmware. The memory state is not captured: recovery restarts from a coherent disk, not from the second before, and a lab whose exercise rests on a running process must restart it. No lab of the published catalogues sets that field to true today.

The machines that outlive their state

A failed provision can leave domains defined on the hypervisor but outside the Terraform state. Reprovisioning on top of them would produce a fleet nobody tracks: dsoxlab refuses instead, and two exit codes say which side gave way.

Code	Meaning
`5`	`provision` found orphan domains and stopped. The message names the command that removes them
`6`	`destroy` could not remove them. Something on the hypervisor still holds them

destroy removes those orphans too, after confirmation, --yes skips it, and exits non-zero if one remains. A destroy that reported success while leaving the machines up is the defect this replaced. Two other codes complete the family: 7, when another dsoxlab command already holds the lock of this catalogue, two clones sharing the same lock because they share the same state, and 8, when provision gave up waiting for hosts that did not answer within the 180-second window.

The stable SSH fragment, and the cache that is not

The ssh_config generated under ~/.cache/dsoxlab/<catalog-id>/ is a cache: it is regenerated on demand, but it is also purged, and anything pointing at it, an Include or an IDE profile, must survive its disappearance. The fragment written to ~/.ssh/config.d/<catalog-id>.conf is the stable one, provided ~/.ssh/config carries an Include of that directory before any Host block. dsoxlab does not write that line for you and warns at every provision while it is missing: without it, the fragment is written and never read, and ssh alma-1.lab keeps failing.

# ~/.ssh/config, at the top of the file
Include ~/.ssh/config.d/*.conf

Using dsoxlab as a provisioner of throwaway VMs

The infrastructure layer knows nothing about labs: provision, destroy, ssh and infra status read the meta.yml and nothing else, no progress database, no score, no discovery. You can therefore use dsoxlab to replay a tutorial on a clean machine before publishing it, reproduce a bug on three distributions, or get a cluster that a container cannot stand in for because the test needs a real kernel, systemd or a firewall. A meta.yml at the root of a directory, and that is all:

repo:
  id: my-stack
  title: "Throwaway VMs"

infra:
  provider: kvm
  network: lab-stack
  cidr: 10.10.90.0/24
  hosts:
    - name: db.lab
      distro: debian13
      ram_mb: 2048
    - name: app.lab
      distro: ubuntu24

No labs/ directory, no repo.category, no Terraform. doctor then classes Terraform, the hypervisor and outbound access as required as soon as infra.hosts is not empty, validate-structure passes, and list-labs prints No labs found., which is not an error. Two limits are accepted: containers do not go through here, runtime.services being declared per lab, and addresses derive from the position in infra.hosts, so a host is always added at the end of the list, libvirt refusing to update an existing network.

Troubleshooting

Each line names a code or a message, its cause, and the gesture the CLI expects. When the case is not here, dsoxlab support --issue files an anonymised diagnostic report, pinned provider version included, in the repository that has to receive it.

Symptom	Cause	Solution
`provision` refuses to start, no key	No key pair under `ssh/` in this clone	`dsoxlab instructor bootstrap`
`provision` exits 5	Orphan domains exist on the hypervisor	Run the `virsh undefine` line printed, or `dsoxlab destroy`, then try again
`provision` exits 8	A host did not answer within 180 seconds	`dsoxlab infra status`, then more vCPU or a longer `DSOXLAB_HOST_READY_TIMEOUT`
Code 7 on a command that writes	Another dsoxlab command holds the catalogue's lock, possibly from another clone	Wait, or close the other terminal, then try again
`ssh alma-1.lab` fails while `dsoxlab ssh` works	`~/.ssh/config` does not carry the `Include` of the stable fragment	Add the `Include` line before any `Host` block
`doctor` refuses the libvirt version	Version below 8.0, never tried	Update libvirt; only `provision` is concerned, a `shell` catalogue is not blocked

Key points

A catalogue declares its machines in infra: and nothing else; the Terraform and cloud-init templates live in the tool.
The kvm network name is bounded to 9 characters after lab-, and provision refuses before doing any work when it overflows.
The SSH key is generated per clone with instructor bootstrap, learners included.
Each provider keeps its own state; dsoxlab infra status is the habit that avoids forgetting a running fleet.
Anything that restricts login targets the ansible account, never student.
5 and 6 say which side the orphans gave way on, 7 is retried, 8 asks for time or vCPU.
A meta.yml without labs/ turns dsoxlab into a provisioner of throwaway VMs, with the same four commands.

Next steps

Incus training: the second local provider, and what an Incus virtual machine changes compared with libvirt.
Installing Incus: the hypervisor the incus provider drives, with the nested virtualisation prerequisite stated from the host's side.