I just spent several days grinding through Incus OS, dissecting every feature one by one. Not out of idle curiosity: I had a precise goal in mind, building a complete private cloud. And I got there. A cluster, Ceph storage, OVN virtual networks, Security Groups, multi-tenancy with quotas and scoped access. A real sovereign IaaS, a scaled-down AWS, on my own machines. The whole recipe is in the Incus section, from the bare server to multi-tenancy.
Except that an empty private cloud is useless. Once the foundation stands, the real question arrives: what do we put on top of it? For me the answer is obvious. I want a software factory, the chain that turns code into deployable artifacts, but secured end to end, multi-tenant and multi-user. The kind of platform I spend my days hardening for other people.
This post is not a deployment tutorial. It is the plan I settled on, and above all a choice I stand behind: I did not build it on Kubernetes. Here is why, and what the blueprint looks like.
What do I call a software factory?
A software factory is a production chain that goes from source code to the deployable artifact, through standardized links: code management, build, tests, packaging, distribution. Making it secure means making sure every link is isolated, traced and provable. You must be able to answer a simple question after the fact: who produced this artifact, from which commit, with which dependencies, and how do you verify it.
This model is not a personal whim. The CNCF and the OpenSSF published a Secure Software Factory Reference Architecture that splits the chain into stages, source, build, materials verification, artifact repository, distribution, admission, with signing and attestations at every step. On the defense side, the DoD Platform One and its Big Bang platform apply the same principles with an in-house forge, a catalog of hardened images and a zero-trust model. And the SLSA framework grades the robustness of the build. I lean on these references, not on my intuition.
Why not Kubernetes?
The current reflex is to stack everything into Kubernetes. Big Bang does exactly that. I say it plainly: for a homelab and for many small and mid-size companies, it is a sledgehammer. You end up maintaining an entire control plane, its operators and its operational debt, before you have even produced the first signed artifact.
What I want first is to compartmentalize workloads that do not trust each other. A build runner executes code I neither wrote nor reviewed: it is the most dangerous zone of the whole chain. A secrets vault holds my signing keys: it is the treasure. Letting them cohabit in the same space is an architectural fault. Incus gives me that isolation as a base building block: its projects separate instances, networks and volumes per tenant, and I mix containers and VMs on the same cluster without imposing a single orchestrator.
As a result, Kubernetes does not disappear, it changes status. It becomes one workload among others, deployed in its own tenant when an application really needs it, not the mandatory foundation of the whole platform. That inversion is the entire bet of the blueprint.
Why one tenant per stage?
The zero-trust principle says that no stage trusts another by default. I translate it directly into Incus objects: one project per sensitive stage, one OVN VPC per project, and ACLs that only open the strictly required flows. It is exactly the segmentation I already validated on a three-tier application, applied this time to the entire factory.
Each stage relies on a 100% free and open source building block, already documented on this site. The stack I picked is deliberately sovereign:
- Source: the Forgejo forge, repositories, review and built-in CI.
- Build: ephemeral runners driven by GARM, one disposable VM or container per job, destroyed afterwards. This is where I borrow the most from an existing project, StepSecurity Harden-Runner, the very component that spotted the tj-actions/changed-files attack in 2025. The catch is that it is coupled to GitHub Actions. So I do not add it, I reproduce its principle on my foundation. Its allowlist-based egress filtering, I enforce at the network level with the build tenant's OVN ACLs: the runner only reaches the forge, the dependency firewall and the registry, nothing else. Its runtime detection of abnormal processes and connections, I hand over to an eBPF sensor inside the runner. Same defense, rebuilt sovereign on Incus. The guide hardening the build environment details the reflexes.
- Materials: an Artifact Firewall that filters incoming dependencies before they can contaminate anything.
- Artifacts: Pulp, with Cosign signing. I clearly prefer it to Harbor on sovereign ground: 100% open source, with no licensing gray zone.
- Evidence: SBOM with Syft, scanning with Trivy or Grype, signing and provenance with Sigstore and SLSA.
- Secrets and identities: the OpenBao vault and a directory for accounts.
None of these bricks needs reinventing. The real work is assembling them cleanly, each within its own perimeter, with the right flows and nothing more.
What path does a commit follow?
To feel how the stages cooperate, I follow a commit end to end. Each step crosses a tenant boundary and adds a verifiable proof.
The developer pushes to Forgejo. A webhook triggers the pipeline, and GARM provisions a disposable runner on Incus, used exactly once. That runner pulls its dependencies through the Artifact Firewall, which blocks or quarantines suspicious versions. The code builds inside this isolated space, then we generate the SBOM, we scan, we sign the artifact and we emit a provenance attestation. The signed deliverable then goes into Pulp, and at deployment time, a policy rejects any artifact that is unsigned or non-compliant.
At the end, every deployed artifact carries complete traceability: which commit, which runner, which dependencies, which signature. That is precisely what the CRA demands and what SLSA grades. And above all, the compromised runner of one tenant can never reach the secrets vault of another: the boundary lives in the network, not just in people's heads.
What is done, and what remains?
I prefer to be honest about the actual state. The private cloud foundation, I built and tested it for real: Incus cluster, storage, OVN networks, ACLs, multi-tenancy, enforced quotas, scoped credentials. That part is solid and documented.
The factory on top is the plan I will roll out stage by stage, reusing each brick in its own tenant. I am not selling you a turnkey platform already in production. I am laying out a coherent target and a path to get there, backed by serious reference architectures. The nuance matters: a credible blueprint is not a finished demo.
Which pitfalls do I anticipate?
A plan is only worth as much as its known blind spots. Four already concern me.
First, the signing PKI is the real treasure. The Cosign keys must live in the vault, never in a runner, and their rotation must be designed from day one. Next, a runner that persists is a compromised runner in the making: I demand truly ephemeral runners, destroyed after every job. Third, OVN segmentation has a known limit: the default-reject of ACLs is buggy on the Incus side, so I write explicit drop rules instead of relying on the implicit reject. Finally, a self-service portal is still missing: Incus is driven by command line and API, and a multi-tenant cockpit in the style of a public cloud remains a layer to build on top. That is the missing link I keep in my sights.
Key takeaways
- A secure software factory isolates, traces and proves every link, from code to the deployed artifact.
- The model comes from solid references: the CNCF Secure Software Factory, Platform One / Big Bang, SLSA.
- I picked Incus as the foundation instead of all-in Kubernetes: native multi-tenancy gives me one tenant per stage, mixing containers and VMs.
- The stack is 100% open source: Forgejo, GARM, Artifact Firewall, Pulp, Cosign, OpenBao, all meshed with the existing guides.
- The private cloud foundation is built and tested; the factory on top will roll out stage by stage, without overselling.
- The real hard points are the signing PKI, the ephemeral runners and the still missing self-service portal.