Debugging an Incus VM stuck at boot (emergency mode)

Read this page in French

Your Incus VM shows as RUNNING in incus list, but incus exec and SSH no longer answer? When the guest agent is dead, the VM operating system never finished booting: it is most likely stuck in emergency mode. The short answer: stop going through incus exec (which needs the agent), capture the serial console with incus console, and if that is not enough, mount the VM disk from the host to read its systemd journal offline. This guide walks that diagnosis end to end, including a real case where hardening bricked the EFI boot.

TL;DR, the three key commands

# 1. Capture the serial console of a stuck VM (forced pty)
timeout 14 script -qec 'incus console <vm>' /dev/null </dev/null 2>&1 | tail -45

# 2. Mount the VM disk (ZFS backend) from the host
incus stop <vm> --force
sudo zfs set volmode=dev <pool>/virtual-machines/<vm>.block
sudo kpartx -av /dev/zd0 && sudo mount -o ro /dev/mapper/zd0p2 /mnt/vm

# 3. Read the failed boot journal offline
sudo journalctl -D /mnt/vm/var/log/journal --list-boots
sudo journalctl -D /mnt/vm/var/log/journal -b <offset> -p err --no-pager

What you will learn

• Tell a dead agent from a plain SSH issue: why a RUNNING VM stays unreachable.
• Capture the serial console of a VM that no longer answers, despite having no real TTY.
• Mount the disk of a stopped VM from the host (ZFS, dir, qcow2) to read its journal.
• Read the offline journalctl of a failed boot with journalctl -D.
• Analyse a real case where a hardening sysctl broke the /boot/efi mount.
• Put the safety nets in place: snapshots, persistent journald, a break-glass root.

Prerequisites

• An Incus host with at least one virtual machine (not a container).
• Root access on the host, which the disk mounting operations require.
• Some knowledge of systemd (targets, units) and Linux partitioning.

Persistent journald: enable it BEFORE the incident

The offline journal method only works if the VM systemd journal is persistent (Storage=persistent, or the /var/log/journal/ directory exists). Otherwise journald keeps everything in RAM and the failed boot journal is lost at reboot. Enable it when you create your lab VMs.

The symptom: RUNNING but unreachable

You restart a VM, then nothing. incus list still shows it started:

incus list <vm>
# +------+---------+------+------+-----------------+-----------+
# | NAME | STATE   | IPV4 | IPV6 | TYPE            | SNAPSHOTS |
# +------+---------+------+------+-----------------+-----------+
# | <vm> | RUNNING |      |      | VIRTUAL-MACHINE | 0         |
# +------+---------+------+------+-----------------+-----------+

Note the missing IPv4 address: the operating system never configured its network. An execution attempt confirms the diagnosis:

incus exec <vm> -- whoami
# Error: VM agent isn't currently running

The RUNNING state in Incus only means QEMU is running the machine. It says nothing about the guest system. Here the agent (incus-agent) never started: the operating system stopped along the way. This is not an SSH problem, it is a boot that never completes.

Dead agent means an unbooted OS

The incus-agent only starts once the guest system is far enough into its boot. If it is absent, there is no point looking at sshd or networking: the VM stopped earlier, often in emergency mode or frozen on a start job.

Why the usual reflexes fail on a VM

On an Incus container, troubleshooting is direct. On a virtual machine, several usual commands fall flat, because they assume a working guest or a real terminal.

Command	On a stuck VM	Why
incus exec <vm>	VM agent isn't currently running	needs the agent, so a booted OS
incus console <vm> --show-log	empty or partial serial log	the serial buffer is not populated like a container log
incus info <vm> --show-log	of little use for a VM	no systemd boot detail
incus console <vm> in a pipe	inappropriate ioctl for device	no real TTY inside a pipe

Glissez pour voir

incus exec died with the agent. --show-log, very useful for a container, does not give the internal boot journal of a VM, so another approach is needed. And piping incus console fails, because the command demands a pseudo-terminal.

Level 1: capturing the serial console

When you simply want to see the screen of the VM (the boot error message), attach the serial console. The trap: incus console demands a pseudo-terminal, so a naive redirection fails. You force a pty with script:

timeout 14 script -qec 'incus console <vm>' /dev/null </dev/null 2>&1 | tail -45

• script -qec '...' /dev/null runs the command in a pty with no file recorded (-q quiet, -e propagates the exit code, -c runs the command).
• timeout 14 cuts after 14 seconds, enough to capture the current screen.
• tail -45 keeps only the last lines, where the final state appears.

What you will read, depending on the failure:

You are in emergency mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, or "exit" to continue bootup.

or a start job that never finishes:

[*** ] A start job is running for /boot/efi (1min 30s / no limit)

or a kernel panic. To detach the interactive console cleanly, press Ctrl+a then q.

The limits of the serial console

The serial console has no scrollback: the [FAILED] and Dependency failed lines scroll past and are lost. And a scripted sulogin login is not reliable (no-echo password entry through the pty). For the complete, reproducible boot detail, move to the offline journal.

Level 2: reading the journal offline by mounting the disk

This is the right reflex. Rather than fighting a blind console, you stop the VM, mount its disk from the host and read its systemd journal calmly. The procedure depends on the storage backend.

Identify the driver first:

incus storage list
# +---------+--------+-------------+---------+
# | NAME    | DRIVER | DESCRIPTION | USED BY |
# +---------+--------+-------------+---------+
# | default | zfs    |             | 12      |
# +---------+--------+-------------+---------+

ZFS (zvol)

On ZFS, the VM disk is a zvol. Once the VM is stopped, that zvol sits at volmode=none: no /dev/zdN device is exposed. You expose it, read its partitions with kpartx, then mount the root read only:

incus stop <vm> --force

# Identify the disk zvol (the .block suffix)
zfs list -t volume | grep <vm>
Z=<pool>/virtual-machines/<vm>.block

# Expose the zvol as a block device
sudo zfs set volmode=dev "$Z"

# kpartx creates the partition mappings (/dev/mapper/zd0p1, zd0p2 and so on)
sudo kpartx -av /dev/zd0

# Mount the root read only (p2 = ext4 root, p1 = EFI vfat)
sudo mkdir -p /mnt/vm
sudo mount -o ro /dev/mapper/zd0p2 /mnt/vm

volmode=dev hides the partitions, hence kpartx

volmode=dev does expose /dev/zd0, but it hides its partitions: you will not get a /dev/zd0p1 automatically. That is exactly why kpartx -av is needed (or volmode=full, which would expose them but stays less predictable). partprobe alone does not always create the nodes.

After the diagnosis, clean up in reverse order:

sudo umount /mnt/vm
sudo kpartx -d /dev/zd0
sudo zfs set volmode=none "$Z"

dir (raw image)

On the dir backend, the disk is a raw image (root.img). You attach it to a loop device with partition scanning (-P):

incus stop <vm> --force

IMG=/var/lib/incus/storage-pools/default/virtual-machines/<vm>/root.img
sudo losetup -fP "$IMG"
losetup -j "$IMG"          # find the assigned loopN (for example /dev/loop0)

sudo mkdir -p /mnt/vm
sudo mount -o ro /dev/loop0p2 /mnt/vm

Cleanup:

sudo umount /mnt/vm
sudo losetup -d /dev/loop0

qcow2

For a qcow2 image, you go through the Network Block Device:

incus stop <vm> --force

sudo modprobe nbd max_part=8
sudo qemu-nbd --connect=/dev/nbd0 <image>.qcow2

sudo mkdir -p /mnt/vm
sudo mount -o ro /dev/nbd0p2 /mnt/vm

Cleanup:

sudo umount /mnt/vm
sudo qemu-nbd -d /dev/nbd0

Once the root is mounted, read the failed boot journal. On a mounted disk the "current boot" is not necessarily the last one: list the boots, then target the offset you want.

# List the recorded boots
sudo journalctl -D /mnt/vm/var/log/journal --list-boots
# IDX BOOT ID    FIRST ENTRY        LAST ENTRY
#  -1 a1b2...    2026-06-29 09:14   2026-06-29 09:31
#   0 c3d4...    2026-06-29 09:32   2026-06-29 09:32   <- the failed boot

# Read the errors of that boot
sudo journalctl -D /mnt/vm/var/log/journal -b 0 -p err --no-pager

# Find the cause directly
sudo journalctl -D /mnt/vm/var/log/journal -b 0 --no-pager \
  | grep -iE 'Dependency failed|Failed to mount|emergency'

That final grep almost always surfaces the chain of causality: a unit that fails, an unsatisfied dependency, then the switch to emergency.

Case study: hardening that bricks the EFI boot

Here is a real, reproducible case that illustrates the whole chain. The offline journal reveals this:

systemd[1]: Mounting boot-efi.mount - /boot/efi...
mount[420]: mount: /boot/efi: unknown filesystem type 'vfat'.
systemd[1]: boot-efi.mount: Mount process exited, code=exited, status=32
systemd[1]: Failed to mount boot-efi.mount - /boot/efi.
systemd[1]: Dependency failed for local-fs.target - Local File Systems.
systemd[1]: Reached target emergency.target - Emergency Mode.

The root of the problem: hardening had set kernel.modules_disabled=1 through a /etc/sysctl.d/ drop-in. That sysctl is a one-way switch: once at 1, no kernel module can be loaded any more, and it cannot go back to 0 without a reboot.

systemd applies sysctls very early at boot, through systemd-sysctl. Its ordering relative to mounting /boot/efi is not guaranteed: both converge on sysinit.target with no strict mutual ordering. If kernel.modules_disabled=1 is applied before mounting /boot/efi triggers loading the vfat module, and if that module is not already loaded, the mount fails. No vfat, no EFI partition mounted, local-fs.target fails, and the system drops into emergency mode.

The lesson: a one-way disabling sysctl such as kernel.modules_disabled must never live in a sysctl.d drop-in applied at boot. Set it late, through a systemd oneshot unit ordered After=local-fs.target (or even after the network), once every needed module is already loaded.

Bonus trap: a locked root makes emergency unreachable

On cloud images, the root account is often locked (* in /etc/shadow). The consequence: emergency mode becomes unreachable, because sulogin refuses an account with no password. You see the prompt but cannot log in. Always plan a break glass: a recovery account, or a documented GRUB password.

Recovery and safety net

Before any risky operation on a VM, take a snapshot. That is your way back:

# Before a risky boot change
incus snapshot create <vm> before-hardening

# Roll back if the VM no longer boots
incus snapshot restore <vm> before-hardening

The trap of a snapshot taken too early

A restore brings the VM back exactly to the snapshot state. If you took the snapshot before installing a package, an audit agent for instance, the restore removes it. Take the snapshot just before the change you are testing, not at the start of the session.

On the access side, never lock root without leaving a way back in. Emergency mode is precisely when you need it:

• a local recovery account with a password,
• or a documented GRUB password to edit the kernel command line at boot.

Prevention: the checklist

The best debugging is the kind you never have to do. Three reflexes avoid most of these lockups:

1. Persistent journald from the start

   On your lab VMs, enable Storage=persistent (or create /var/log/journal/) at creation time. Without it, the failed boot journal goes up in smoke at reboot.

2. Test boot changes on a disposable VM

   Anything touching boot (/etc/fstab, the kernel command line, a one-way sysctl) gets tested on a disposable VM protected by a snapshot, never directly on a machine that matters.

3. Find the culprit through the trace, not through blind reboots

   Prefer a dry run of your configuration (the converged state without rebooting) to spot a dangerous change before it bricks the boot, rather than restarting at random.

Key points

• A RUNNING VM with no agent has not finished booting: it is not an SSH problem.
• incus exec needs the agent; on a stuck VM it returns VM agent isn't currently running.
• For a VM, --show-log gives an often empty serial log: capture the serial console (incus console, detach with Ctrl+a q), forced into a pty through script.
• The right reflex: stop the VM, mount its disk from the host (ZFS through volmode=dev plus kpartx, dir through losetup -fP, qcow2 through qemu-nbd) and read the journal with journalctl -D.
• volmode=dev hides the zvol partitions, hence kpartx.
• A one-way sysctl such as kernel.modules_disabled=1 set too early can break a mount (/boot/efi in vfat) and cause emergency mode.
• Snapshots, persistent journald and a break-glass root are your three safety nets.

FAQ: common questions about debugging an Incus VM

Why does incus exec not work on my virtual machine?

incus exec goes through the guest agent, which only starts once the operating system has booted. If the VM is stuck at boot or in emergency mode, the agent is dead and the command returns Error: VM agent isn't currently running. That is not an SSH problem but a boot that never completes.

How do you read the journal of a powered-off Incus VM?

You mount the VM disk from the host then read its journal with journalctl -D. After incus stop --force, expose the disk (ZFS: volmode=dev plus kpartx; dir: losetup -fP; qcow2: qemu-nbd), mount the root read only, then run journalctl -D /mnt/vm/var/log/journal --list-boots.

Does incus console --show-log work on a virtual machine?

Not usefully. On a container, --show-log shows the console log. On a virtual machine that serial log is often empty and holds no systemd boot detail. You attach the interactive serial console with incus console instead, detaching with Ctrl+a then q, and read the journal offline.

How do you mount the disk of an Incus VM stored on ZFS?

After incus stop --force, the disk is a zvol in volmode=none. You expose it with zfs set volmode=dev on the .block dataset, which creates /dev/zd0 but hides its partitions. You make them visible with kpartx -av /dev/zd0, then mount /dev/mapper/zd0p2 read only.

Why does my Incus VM drop into emergency mode at boot?

Emergency mode happens when local-fs.target fails, often an impossible mount from a wrong fstab entry or a missing partition. A classic case: kernel.modules_disabled=1 applied too early prevents the vfat module, breaking the /boot/efi mount. The offline journal reveals the cause.

Next steps

• Securing Incus: the resource limits and isolation that stop a drifting instance taking the host with it.
• Managing instances: the console commands this guide relies on, in their ordinary context.
• Incus OS without a shell: a system with no shell, where diagnosis goes entirely through the serial console and the API.

×

📖 Voir la documentation →