Maintaining self-hosted runners

Self-hosted runners require regular maintenance to stay fast and secure. This guide covers the essential operations: updates, monitoring, cleanup and scaling.

What you will learn

Keep up to date the runner, the OS and the tools
Supervise the runners with health checks and Prometheus
Clean up Docker, the work directory and the local caches
Grow the fleet, manually or through auto-scaling
Provide high availability to avoid single points of failure

Updates

Three components need a different kind of attention: the runner (managed by GitHub), the OS and the build tools (both on you).

Automatic runner updates

The runner updates itself when GitHub ships a new version. During a job, if an update is available, it is applied before the run starts.

Check the version:

./run.sh --version
# or
cat /opt/actions-runner/.runner | jq '.agentVersion'

OS updates

The host system, on the other hand, is updated by nobody: that is your job. The risk is not the update itself but the reboot it may require in the middle of a job. unattended-upgrades applies security patches automatically; dpkg-reconfigure opens an interactive dialog where you choose whether updates install on their own.

# Ubuntu/Debian
sudo apt update && sudo apt upgrade -y

# With a scheduled reboot
sudo apt install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades

Tool updates

A GitHub-hosted runner starts from a regularly rebuilt image. A self-hosted runner keeps the versions installed the day it went into service: the gap with what your workflows assume widens silently. Node.js, Docker and kubectl are the three tools that fall behind the fastest.

# Node.js through nvm
nvm install --lts
nvm alias default lts/*

# Docker
sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io

# kubectl: download the binary AND its official checksum
KUBECTL_VERSION="$(curl -L -s https://dl.k8s.io/release/stable.txt)"
curl -LO "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl"
curl -LO "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl.sha256"

# Refuse the installation if the fingerprint does not match
echo "$(cat kubectl.sha256)  kubectl" | sha256sum --check
sudo install kubectl /usr/local/bin/

The sha256sum --check verification prints kubectl: OK when the fingerprint matches. If it prints FAILED, the downloaded binary is not the one published by the project: do not install it.

Monitoring

A runner going down without an alert blocks the whole CI silently. Monitoring makes the state of the machine and the availability of the runner visible.

System metrics

These commands give the state of the machine right now. The last one is the most revealing: Runner.Listener is the process keeping the connection with GitHub open and waiting for jobs. If it does not show up, the runner is displayed as offline in the interface, even if the machine answers perfectly to everything else.

# CPU, memory, disk
top -bn1 | head -20
free -h
df -h

# The runner process
ps aux | grep Runner.Listener

Health check script

This script checks the three most frequent causes of failure: the stopped process, the full disk and the broken connection to the GitHub API. Its exit codes follow the convention of monitoring probes (0 for healthy, 1 for warning, 2 for critical), which lets you plug it as it is into most supervision systems.

#!/bin/bash
RUNNER_DIR="/opt/actions-runner"

# Check that the runner is running
if ! pgrep -f "Runner.Listener" > /dev/null; then
    echo "CRITICAL: Runner not running"
    exit 2
fi

# Check the free space of the volume holding the work directory
DISK_USAGE=$(df -h "$RUNNER_DIR" | awk 'NR==2 {print $5}' | sed 's/%//')
if [ "$DISK_USAGE" -gt 90 ]; then
    echo "WARNING: Disk usage at ${DISK_USAGE}%"
    exit 1
fi

# Check the connection to GitHub
if ! curl -s --connect-timeout 5 https://api.github.com > /dev/null; then
    echo "CRITICAL: Cannot reach GitHub API"
    exit 2
fi

echo "OK: Runner healthy"
exit 0

Prometheus and Grafana

The health check answers yes or no; Prometheus keeps the history and lets you see a saturation coming before it blocks a job. node_exporter is the agent exposing the system metrics of the machine on port 9100, which Prometheus then scrapes. Started as a container, it only sees the container by default: the /proc and /sys directories of the host are therefore mounted read-only, and the --path.procfs and --path.sysfs options tell it where to read them.

services:
  node-exporter:
    image: prom/node-exporter:v1.12.1@sha256:1b4e4438faca4dd7e001dd445d161a4a2091b0fededa84093b3a8dfeae1f1be0
    ports:
      - "9100:9100"
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
    command:
      - "--path.procfs=/host/proc"
      - "--path.sysfs=/host/sys"

Useful metrics to watch:

node_cpu_seconds_total: CPU usage
node_memory_MemAvailable_bytes: available memory
node_filesystem_avail_bytes: disk space
Custom metrics on the jobs that ran

Alerting

Two alerts cover the essentials: the unreachable runner and the disk close to saturation. The for: 5m avoids waking someone up for a plain service restart, and the 5e9 threshold corresponds to 5 GB of free space left, roughly one large build of headroom.

# alertmanager rules
groups:
  - name: runner-alerts
    rules:
      - alert: RunnerDown
        expr: up{job="runner"} == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Runner {{ $labels.instance }} is down"

      - alert: RunnerDiskFull
        expr: node_filesystem_avail_bytes{mountpoint="/"} < 5e9
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Runner disk space low"

Cleanup

On a persistent runner, builds and images pile up until the disk is full. A scheduled cleanup keeps the machine healthy.

Docker cleanup

Each prune targets a different resource and none of them is reversible. The until=24h filter spares the images created in the last 24 hours, so the build cache of the day. The last line is deliberately left commented out: docker system prune -af --volumes also removes every unused volume, named volumes included, and therefore the data of the test services running on the machine.

#!/bin/bash
# Unused images
docker image prune -af --filter "until=24h"

# Stopped containers
docker container prune -f

# Orphan volumes
docker volume prune -f

# Unused networks
docker network prune -f

# Everything at once (aggressive)
# docker system prune -af --volumes

Schedule it with cron:

0 3 * * * root /opt/scripts/docker-cleanup.sh >> /var/log/docker-cleanup.log 2>&1

Cleaning the work directory

The _work directory holds one subdirectory per cloned repository and is never purged by the runner. _diag accumulates the run logs, one file per job. Schedule this cleanup at a quiet hour: deleting _work while a job is running makes that job fail.

#!/bin/bash
RUNNER_DIR="/opt/actions-runner"
WORK_DIR="$RUNNER_DIR/_work"

# Remove the work directories older than 7 days
find "$WORK_DIR" -type d -mtime +7 -exec rm -rf {} \;

# Remove the old logs
find "$RUNNER_DIR/_diag" -name "*.log" -mtime +30 -delete

Cleaning the actions cache

Do not confuse two things here. The cache managed by the actions/cache action is stored at GitHub, not on your machine, and is purged from the repository interface. The directories below are the local caches of the npm, pip and Maven package managers: wiping them breaks nothing, the next build rebuilds them at the price of a full download.

# The actions/cache cache is managed by GitHub
# But the local cache (npm, pip, etc.) can pile up

rm -rf ~/.npm/_cacache
rm -rf ~/.cache/pip
rm -rf ~/.m2/repository

Scaling

When the queue grows, you have to add runners: by hand for a small fleet, automatically as soon as the volume grows.

Manual scaling

An extra runner is configured exactly like the first one. Two details matter: the registration token taken from the repository or organisation settings expires after one hour, and the labels alone decide which jobs the machine will accept.

# On a new machine
./config.sh --url https://github.com/ORG/REPO \
            --token TOKEN \
            --labels linux,x64,docker \
            --name runner-$(hostname)
./run.sh

Auto-scaling with Actions Runner Controller

Actions Runner Controller (ARC) is a Kubernetes operator creating and destroying runner pods on demand. The HorizontalRunnerAutoscaler reacts to the workflowJob events sent by GitHub: a job enters the queue, a runner appears. The scaleDownDelaySecondsAfterScaleOut enforces 300 seconds before any scale down, which avoids destroying a machine that has just been created. The example relies on the actions.summerwind.dev API, the one of the historical ARC mode.

apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: runner-autoscaler
spec:
  scaleTargetRef:
    name: runner-deployment
  minReplicas: 2
  maxReplicas: 20
  scaleUpTriggers:
    - githubEvent:
        workflowJob: {}
      duration: "30m"
  scaleDownDelaySecondsAfterScaleOut: 300

Cloud scaling (AWS)

Without Kubernetes, the fleet becomes a classic autoscaling group. Everything rests on the launch_template: it is the piece that must carry the runner registration at boot, otherwise the created instance will never announce itself to GitHub and will stay billed for nothing. The 300-second cooldown prevents the policy from adding two instances back to back before seeing the effect of the first one.

resource "aws_autoscaling_group" "runners" {
  name                = "github-runners"
  desired_capacity    = 2
  min_size            = 1
  max_size            = 10
  vpc_zone_identifier = var.subnet_ids

  launch_template {
    id      = aws_launch_template.runner.id
    version = "$Latest"
  }

  tag {
    key                 = "Name"
    value               = "github-runner"
    propagate_at_launch = true
  }
}

resource "aws_autoscaling_policy" "scale_up" {
  name                   = "scale-up"
  scaling_adjustment     = 2
  adjustment_type        = "ChangeInCapacity"
  cooldown               = 300
  autoscaling_group_name = aws_autoscaling_group.runners.name
}

High availability

A single runner is a point of failure: if it goes down, the CI stops. Several reflexes remove that risk.

Multiple runners

GitHub assigns a job to the first free runner whose labels cover those requested. By naming a set of labels rather than a specific machine, you let the service arbitrate: losing a runner slows the queue down, it no longer blocks it.

jobs:
  build:
    runs-on: [self-hosted, linux]
    # GitHub will pick an available runner among those that match

Geographical spread

Region labels let you bring a job closer to the resources it handles: image registry, target cluster or database. Keep in mind that a label is a plain string declared when the runner registers: GitHub does not check that it matches a real zone.

jobs:
  build-eu:
    runs-on: [self-hosted, linux, eu-west-1]

  build-us:
    runs-on: [self-hosted, linux, us-east-1]

Falling back to GitHub-hosted

runs-on accepts an expression, which allows a switch without touching the workflow file. Flipping the USE_SELF_HOSTED variable in the repository settings is enough: for the duration of a maintenance window, the jobs go back to the GitHub-hosted runners, billed by the minute.

jobs:
  build:
    # Try self-hosted first, fall back to GitHub-hosted
    runs-on: ${{ vars.USE_SELF_HOSTED == 'true' && 'self-hosted' || 'ubuntu-24.04' }}

Maintenance checklist

So that nothing is forgotten, here are the operations to carry out, sorted by frequency, from the automated daily tasks to the quarterly review.

Daily (automated)

None of these three tasks should depend on a human being present: they run in cron or in a systemd timer, otherwise they are skipped the day you are on holiday.

Health checks every 5 minutes
Docker cleanup overnight
Log rotation

Weekly

A ten-minute review is enough to spot a drift before it becomes an outage: a disk filling up regularly or an alert repeating itself deserves a fix, not an acknowledgement.

Check the OS updates
Check the disk space
Review the alerts of the week

Monthly

The monthly slot handles what moves slowly: tool versions, access rights and build duration. A build time doubling over three months goes unnoticed day to day but stands out in a monthly comparison.

Tool updates (Node, Python, etc.)
Audit of the access to the runners
Performance review (build time)
Check the token rotation

Quarterly

At that rhythm, it is no longer about fixing but about deciding: does the current architecture still handle the load, and is the cost of the fleet still justified against hosted runners? It is also the moment to check that a full return to service really works.

Major OS upgrade if needed
Review of the runner architecture
Disaster recovery test
Cost optimisation

Key points

The runner updates itself; on you: the OS and the tools used by your workflows.
A regular health check (process, disk, GitHub connection) catches failures before they block the CI.
Cleaning up Docker and _work is mandatory: without it, the disk fills up within days.
Auto-scaling (ARC or cloud) matches the fleet to the load, with no idle runners billed for nothing.
Several runners per label remove the single point of failure; a GitHub-hosted fallback covers the peaks.

Next steps

act: reproducing the behaviour of a runner locally to validate an image or a toolchain before deploying it to the fleet.
actionlint: rejecting invalid workflows before they consume the time of your machines.