GPU in Incus: passing an NVIDIA card to a container or a VM

Read this page in French

Running inference or GPU compute inside an Incus instance, without dedicating a whole machine, is exactly what the gpu device allows. For a container, Incus shares the host GPU and injects the NVIDIA userspace by itself (nothing to install inside); for a virtual machine, it is an exclusive PCI passthrough. This guide covers both, including the driver version trap. Tested with an NVIDIA H100 on Incus 7.0.

What you will learn

• The difference between a container (sharing) and a VM (exclusive passthrough).
• Attach an NVIDIA GPU to a container with nvidia.runtime.
• Filter the right GPU on a multi-card machine.
• The driver version traps.

Prerequisites

• A host with an NVIDIA GPU and its kernel driver installed (nvidia-smi works on the host).
• Incus installed: see installing Incus.
• For automatic injection into containers: nvidia-container-toolkit (the libnvidia-container package) on the host.

Container or VM: two different mechanisms

This is the point to grasp first. The gpu device does not do the same thing depending on the instance type.

Type	Mechanism	Sharing	Driver inside the instance
Container	Shares the host /dev/nvidia* devices	Yes, across instances	Injected by nvidia.runtime
Virtual machine	PCI passthrough (VFIO)	No, exclusive to the VM	Installed inside the VM

Glissez pour voir

For inference shared across several environments, the container is the more flexible route. For strong isolation or a different guest operating system, the virtual machine takes the GPU exclusively.

Passing a GPU to a container

The elegant part of Incus: with nvidia.runtime=true, it mounts the host NVIDIA userspace (libraries and nvidia-smi) into the container. No CUDA package to install inside.

1. Launch a container with the NVIDIA runtime enabled:

   incus launch images:debian/13 gpu-lab -c nvidia.runtime=true

2. Attach the GPU with a gpu device:

   incus config device add gpu-lab gpu0 gpu
   # Device gpu0 added to gpu-lab

3. Check inside the container, with nothing installed in it:

   incus exec gpu-lab -- nvidia-smi -L

   GPU 0: NVIDIA H100 PCIe (UUID: GPU-ada5ffa8-3c66-...)

The host GPU is seen as it is inside the container, with the /dev/nvidia0 and /dev/nvidiactl nodes exposed. Since this is sharing, several containers can receive the same GPU.

nvidia.runtime needs libnvidia-container on the host

Automatic userspace injection relies on libnvidia-container (shipped by nvidia-container-toolkit, the same one Docker uses). Without it on the host, nvidia.runtime=true mounts nothing and nvidia-smi fails inside the container with a library error. On the test host: nvidia-container-toolkit 1.19.0.

Targeting the right GPU

On a machine with several cards, the gpu device without a filter exposes all of them. You restrict it with filtering properties.

# by card identifier
incus config device add gpu-lab gpu0 gpu id=0

# by PCI address
incus config device add gpu-lab gpu0 gpu pci=0000:81:00.0

You can also filter by vendorid and productid. This is essential to dedicate one card to an instance and leave others free for the host or for other instances.

GPU in a virtual machine

For a virtual machine, the gpu device triggers a PCI passthrough: the card is detached from the host and attached to the VM, exclusively. The host can no longer use it while the VM runs.

incus launch images:debian/13 gpu-vm --vm
incus config device add gpu-vm gpu0 gpu

The VM installs its own driver

Unlike a container, nvidia.runtime has no effect on a virtual machine: the guest sees a real PCI device and has to install the matching NVIDIA driver itself. Passthrough also requires IOMMU enabled on the host (intel_iommu=on or amd_iommu=on).

The driver version trap

The most frequent problem in containers is a version mismatch between the host kernel driver and the injected userspace. Since nvidia.runtime mounts the host userspace, the two are aligned by construction here. If, however, you install NVIDIA packages by hand inside the container, they can conflict with the host kernel driver (Failed to initialize NVML: Driver/library version mismatch). The rule: in a container, let nvidia.runtime manage the userspace and install no driver inside.

Key points

• The gpu device means sharing in a container and an exclusive PCI passthrough in a virtual machine.
• In a container, nvidia.runtime=true injects the NVIDIA userspace: nothing to install inside.
• Injection requires libnvidia-container (nvidia-container-toolkit) on the host.
• Target a specific card with id=, pci= or vendorid and productid.
• In a virtual machine: the driver goes in the guest, and IOMMU is required on the host.

FAQ: common questions about GPU in Incus

How do you pass an NVIDIA GPU to an Incus container?

You launch the container with nvidia.runtime=true, then add a gpu device with incus config device add. Incus injects the host NVIDIA userspace and nvidia-smi works inside the container without installing anything in it.

What is the difference between GPU in a container and in a VM with Incus?

In a container the GPU is shared: several instances use it and the userspace is injected by nvidia.runtime. In a virtual machine it is an exclusive PCI passthrough: the card is detached from the host, and the guest has to install its own NVIDIA driver.

Why does nvidia-smi fail inside an Incus container?

Most often because libnvidia-container is not installed on the host: without it, nvidia.runtime injects no userspace. The other cause is a driver installed manually inside the container, conflicting with the host kernel driver version.

How do you target a specific GPU on a multi-card machine with Incus?

You filter the gpu device with a property: id for the card identifier, pci for the PCI address, or vendorid and productid. Without a filter, the gpu device exposes every card of the host to the instance.

Next steps

• Managing instances: the devices and configuration around the card you just attached.
• Incus OS without a shell: the immutable system where the NVIDIA driver version follows the atomic OS updates.
• Securing Incus: what exposing a host device to an instance implies.

×

📖 Voir la documentation →