⏱️ Lectura: 11 min
In his first week as a solutions engineer at Oxide Computer, in late 2024, Matthew Sanabria received two documents to get started: a pull request submitted by a customer with a hand-built Rancher node driver, and the draft of RFD 493, “Initial Kubernetes Integrations.” From there came, in under two years, the three official integrations that today allow running Kubernetes on Oxide: a driver for Rancher, an infrastructure provider for Omni, and a Cluster API provider.
📑 En este artículo
Oxide builds complete compute, network, and storage racks with its own API instead of the traditional BMCs found in commercial servers. That design fits naturally with Kubernetes, which defines the infrastructure behavior it expects through standard extension points. What was missing was the software connecting both worlds, and that’s the work Oxide documented on its blog.
TL;DR
- Oxide Computer published how it built its three Kubernetes integrations: Rancher, Omni, and Cluster API.
- Matthew Sanabria, Oxide’s first Solutions Software Engineer, started the project in late 2024.
- The Rancher node driver was the first integration: it translates Rancher actions into calls to the Oxide API.
- The infrastructure provider for Omni was built in seven weeks, in time for KubeCon NA 2025.
- A bug in Talos Linux (siderolabs/talos#11948) prevented reading Oxide’s user-data due to a filesystem issue.
- The temporary fix: padding the user-data file with comments to force the ISO 9660 superblock.
- The third integration is a Cluster API (CAPI) provider, declarative and with no third-party dependency.
- All three integrations now have official guides published by Oxide.
Introduction: What Running Kubernetes on Oxide Means
Kubernetes was designed from the start to not depend on any particular cloud provider. It defines standard extension points (cloud provider interfaces, storage drivers, infrastructure providers) so that any platform can implement them. Oxide, for its part, exposes every hardware operation (creating an instance, attaching a disk, configuring the network) through a documented HTTP API. When those two pieces fit together, running Kubernetes on Oxide becomes an integration problem, not a reinvention one.
The problem, until late 2024, was that this integration didn’t officially exist yet. Customers who wanted Kubernetes on Oxide racks had to figure it out on their own, like the one who ended up submitting the Rancher driver pull request.
What Happened
Sanabria was hired as Oxide’s first Solutions Software Engineer, a role focused on building software to solve concrete customer problems, not abstract research. His first task was to simplify deploying and operating Kubernetes on Oxide.
Instead of designing integrations in a vacuum, the team followed the problems customers ran into as they moved from provisioning clusters to operating real workloads. Different provisioning flows led to Rancher, Omni, and Cluster API. Operating clusters exposed infrastructure reconciliation needs. Exposing applications revealed networking gaps. And stateful workloads exposed storage limitations.
That order (provision, operate, expose, persist) is the one Oxide uses to explain what it worked on first and what’s still missing.
Context and History
The Rancher Node Driver
A Rancher node driver is an executable plugin that teaches Rancher how to create and manage virtual machines on a particular infrastructure platform. Oxide’s driver translates those operations (creating, listing, deleting an instance) into calls to the Oxide API.
Sanabria had never used Rancher or worked with a node driver, so reviewing the customer’s contribution meant learning both at once. After confirming the implementation worked, the team merged the pull request, added continuous integration and documentation, and published the first official version. It was Oxide’s first Kubernetes integration, and it already had a customer running it in production before it was even published.
The Infrastructure Provider for Omni
Omni, from Sidero Labs, is a control plane that provisions and manages Kubernetes clusters on Talos Linux, an immutable Linux distribution built specifically to run Kubernetes. Omni connects to different platforms through infrastructure providers: programs that create Talos Linux instances and register them with Omni.
With KubeCon North America 2025 just weeks away, Oxide saw the opportunity to build that provider together with Sidero Labs and showcase it at a joint event. The team had seven weeks to complete it before the Oxide+Sidero event at KubeCon.
The Cluster API Provider
Cluster API (CAPI) is the Kubernetes API maintained by SIG Cluster Lifecycle for creating, scaling, upgrading, and deleting clusters declaratively using Kubernetes custom resources. Unlike Rancher or Omni, it doesn’t depend on a third-party platform: each infrastructure provider implements the resources that tell CAPI how to create and delete virtual machines on a specific platform.
Oxide already intended to build this provider from the first draft of RFD 493, but early on, customer demand and engineering capacity didn’t justify the investment. It ended up being the third integration published, after confirming that no single approach covered all of its customers’ workflows.
Technical Details and Performance
The most cited technical detail of the project is a compatibility issue between Oxide’s cloud-init and Talos Linux’s filesystem probe. Oxide uses a FAT12 filesystem for the cloud-init user-data disk, not ISO 9660. But Talos’s probe only tried to read an ISO 9660 superblock on the NoCloud configuration disk, and if that read failed, it stopped instead of trying other formats like VFAT or MS-DOS. The result: Talos never read Oxide’s user-data with the configuration needed to join Omni.
⚠️ Heads up: the fix for this bug didn’t land in time for KubeCon. The temporary workaround Oxide published was padding the user-data file with comments until the disk reached a size large enough for the probe to use the ISO 9660 superblock.
The bug was publicly documented in siderolabs/talos#11948, part of a broader set of reports Oxide opened in siderolabs/omni#1633. Sidero Labs responded quickly to the reports, something Oxide attributes to its own RFD (Request for Discussion) process, the internal mechanism the company uses to document design decisions before writing code.
The Cluster API provider follows a different model: instead of a binary that translates individual commands, it defines Kubernetes resources that describe the cluster’s desired state. A typical manifest looks like this:
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: prod-oxide-01
namespace: default
spec:
clusterNetwork:
pods:
cidrBlocks: ["10.244.0.0/16"]
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: OxideCluster
name: prod-oxide-01The exact resource name (OxideCluster) illustrates the pattern followed by all CAPI infrastructure providers (AWSCluster, VSphereCluster, etc.); confirm the actual name in the documentation Oxide publishes alongside the provider. CAPI continuously reconciles that manifest: if someone manually deletes a machine, the controller recreates it so the actual state matches the declared one. That’s the key difference from a node driver, which only acts when Rancher explicitly requests it.
flowchart TD
A["Kubernetes"] --> B["Rancher node driver"]
A --> C["Omni infrastructure provider"]
A --> D["Cluster API provider"]
B --> E["Oxide API"]
C --> E
D --> E
E --> F[("Oxide Racks")]How to Choose Between the Three
| Integration | When to Use It | Advantage | Limitation |
|---|---|---|---|
| Rancher node driver | You already manage clusters with Rancher Manager | Installs as a plugin from the UI, no YAML required | Depends on having Rancher Manager running and up to date |
| Omni infrastructure provider | You run Talos Linux and want centralized management | Takes advantage of Talos’s immutable model and Omni’s control plane | Requires applying the temporary user-data workaround until the Talos fix ships |
| Cluster API provider | You already use CAPI on other clouds and want the same flow on Oxide | Standard declarative API, no third-party platform involved | Requires defining and maintaining more custom resources per cluster |
💡 Tip: if you already manage clusters on other clouds with Cluster API, Oxide’s provider lets you keep the same workflow (same clusterctl, same manifests) instead of learning a new tool just for your on-premise hardware.
Getting Started
The most direct way to try the Cluster API provider is to install clusterctl, the project’s official CLI, and point it at the Oxide provider.
Linux
curl -L https://github.com/kubernetes-sigs/cluster-api/releases/latest/download/clusterctl-linux-amd64 -o clusterctl
chmod +x clusterctl
sudo mv clusterctl /usr/local/bin/clusterctl
clusterctl versionmacOS
curl -L https://github.com/kubernetes-sigs/cluster-api/releases/latest/download/clusterctl-darwin-amd64 -o clusterctl
chmod +x clusterctl
sudo mv clusterctl /usr/local/bin/clusterctl
clusterctl versionWindows (PowerShell)
curl.exe -L https://github.com/kubernetes-sigs/cluster-api/releases/latest/download/clusterctl-windows-amd64.exe -o clusterctl.exe
.\clusterctl.exe versionWith the binary installed, initialize the corresponding infrastructure provider and confirm it appears in the list of active providers:
clusterctl init --infrastructure oxide
clusterctl config repositories | grep oxideIf you prefer Rancher, the driver is added from Cluster Management > Drivers > Node Drivers in the UI, pointing to the URL of the binary Oxide publishes. For Omni, you need to register the infrastructure provider following the specific guide Oxide links from its original post; that’s also where you’ll find the details on how to apply the temporary user-data workaround while the Talos fix isn’t available yet.
Impact and Analysis
What matters in this case isn’t a single integration but the method: Oxide didn’t design three ways to run Kubernetes just because. Each customer arrived with a different workflow (Rancher, Omni, or CAPI), and none of the three covered everyone. For infrastructure teams in Latin America evaluating on-premise or sovereign hardware for cost, regulatory, or latency reasons, this matters because it reduces the risk of getting locked into a single orchestration tool when choosing a hardware provider.
It’s also a case study in how a hardware company competes in the Kubernetes ecosystem without building its own distribution: instead, Oxide invested in integrating with the tools its customers already used (Rancher, Sidero Labs, SIG Cluster Lifecycle), including collaborating on reporting and fixing bugs in third-party projects like Talos Linux.
What’s Next
Oxide’s own post leaves the following areas open: operating clusters exposed infrastructure reconciliation needs that are still being worked out, exposing applications revealed gaps in Oxide’s networking model (relevant for anyone needing a CNI or native load balancing), and stateful workloads exposed persistent storage limitations, an area where future work on CSI drivers would be expected. Oxide doesn’t give concrete dates for these pieces, but points to them as the natural continuation of the same customer-driven process that produced the three provisioning integrations.
📖 Summary on Telegram: See summary
Try it yourself: install clusterctl, run clusterctl init --infrastructure oxide, and confirm with clusterctl config repositories | grep oxide that the Oxide provider shows up as active before spinning up a real cluster.
Frequently Asked Questions
What Is Oxide Computer?
It’s a company that builds complete compute, network, and storage racks with its own API built into the hardware, instead of relying on traditional third-party BMCs.
What Is a Rancher Node Driver?
An executable plugin that teaches Rancher how to create, list, and delete virtual machines on a specific infrastructure platform, in this case Oxide.
What’s the Relationship Between Omni and Talos Linux?
Omni is Sidero Labs’ control plane that provisions and manages Kubernetes clusters running on Talos Linux, an immutable Linux distribution designed for Kubernetes.
What Is Cluster API, and How Does It Differ from Rancher or Omni?
It’s a declarative Kubernetes API for managing clusters with custom resources, without depending on a third-party platform like Rancher or Omni.
Why Couldn’t Talos Linux Read Oxide’s User-Data?
Because Oxide uses a FAT12 filesystem for that disk, and Talos’s probe only tried to read an ISO 9660 superblock, without trying other formats if that read failed.
Are the Kubernetes on Oxide Integrations Free to Use?
Oxide publishes them as part of its official product guides for customers running Oxide hardware; development was coordinated publicly with Sidero Labs on GitHub.
References
- Oxide Computer: original post, “Kubernetes on Oxide: How Customer Needs Shaped Our Integrations”.
- siderolabs/talos#11948: issue where Sidero Labs documented the filesystem detection bug in the user-data.
- siderolabs/omni#1633: reports on the integration between Oxide and Omni.
- Cluster API (SIG Cluster Lifecycle): official documentation for the CAPI project.
- Rancher Manager Docs: official documentation for Rancher and its node drivers.
📱 Like this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day.
0 Comments