NVKE: NVIDIA’s Factory for Bare Metal Kubernetes

⏱️ Reading time: 11 min

Installing NVIDIA GPU Operator assumes you already have a Kubernetes cluster running, with nodes ready and a control plane in place. NVIDIA just released, in preview, a project that tackles the problem one layer down: GPU management in Kubernetes from scratch, on bare metal machines that don’t yet have a control plane or nodes joined to each other.

📑 En este artículo

1. TL;DR
2. What Is NVKE?
3. Why GPU Management in Bare Metal Kubernetes Matters
4. How NVKE Works Under the Hood
   1. The Seed Cluster and the Management Cluster
   2. Workload Clusters and Their Node Pools
   3. NICo, the Only Compatible Infrastructure Provider
   4. Node Identity with SPIFFE/SPIRE
5. Practical Examples and Getting Started
6. Real-World Use Cases
7. Common Mistakes and Best Practices
8. NVKE vs. NVIDIA GPU Operator
9. Going Deeper: Node Identity and the Link with NVKR
10. Frequently Asked Questions
    1. Does NVKE Replace NVIDIA GPU Operator?
    2. Can NVKE Be Used in Production Today?
    3. What Infrastructure Providers Does NVKE Support?
    4. What Does nvkectl Do Within NVKE?
    5. Does NVKE Run on macOS or Windows?
11. References

The project is called NVKE, short for NVIDIA Kubernetes Engine. Its own GitHub repository describes it as a ‘self-hosting factory for Kubernetes clusters’: a factory that assembles entire clusters, not just a package of GPU drivers.

TL;DR

• NVKE is NVIDIA’s preview-stage management plane for creating Kubernetes clusters on bare metal GPUs.
• nvkectl spins up a temporary seed cluster, forms the management cluster, and from there creates independent workload clusters.
• NICo is, in this preview, the only infrastructure provider compatible with NVKE.
• Nodes authenticate via SPIFFE/SPIRE and their TPM before receiving cluster join credentials.
• GPU Operator installs drivers on an existing cluster; NVKE builds the entire cluster from scratch.

What Is NVKE?

NVKE is NVIDIA’s platform for GPU management in Kubernetes on your own bare metal infrastructure, without depending on a cloud provider. It works as a management plane: it’s installed once, and from there a platform team creates and operates multiple independent workload clusters.

The project lives in preview state, which means NVIDIA doesn’t recommend it yet for production workloads: its APIs, configuration, and behavior can change between releases, so any evaluation should be done in a disposable environment.

NVKE is not a Kubernetes distribution or a variant of Cluster API. It is, in the project’s own words, a self-hosting cluster factory: a disposable seed cluster first spins up a management cluster, and that management cluster then creates and operates, on demand, as many workload clusters as the platform team needs.

Why GPU Management in Bare Metal Kubernetes Matters

Buying or renting GPUs outside a public cloud forces you to solve a problem that clouds give away for free: turning a warehouse full of servers into several isolated Kubernetes clusters, each with its own control plane, without the infrastructure team having to build that automation by hand.

That’s exactly what NVKE tackles. An infrastructure operator installs the management plane once across its fleet of GPU machines, and from there every internal customer, every research project, or every machine learning team gets its own workload cluster, with its own etcd, in minutes instead of weeks of tickets filed against the datacenter team.

GPU management in bare metal Kubernetes matters in particular for universities, labs, and companies that already bought the GPU hardware and don’t want to pay a public cloud’s markup just to get the orchestration layer on top.

How NVKE Works Under the Hood

An NVKE installation has two kinds of Kubernetes clusters. The management cluster runs the management plane and infrastructure controllers; nvkectl forms it using a temporary seed cluster, and installing the management plane services is a separate step. Workload clusters, on the other hand, run each customer’s applications and are created and operated through NICo, Cluster API, and a node agent.

flowchart TD
  A["Temporary seed cluster"] --> B["NVKE management cluster"]
  B --> C["NICo: Infra Controller"]
  C --> D(("Bare metal nodes"))
  B --> E["Workload cluster 1"]
  B --> F["Workload cluster 2"]
  subgraph "Workload clusters"
  E
  F
  end

The Seed Cluster and the Management Cluster

The seed cluster exists only to bootstrap: once the management cluster is up, the seed is discarded. From there, the platform team installs on the management cluster the services that expose the NVKE API and the nvkectl CLI, which is the entry point for requesting new clusters.

Workload Clusters and Their Node Pools

Each workload cluster is requested through the NVKE API or via nvkectl; NVKE provisions the machines through NICo and forms the cluster with its own control plane. The same plane then lets you create, scale, and delete node pools, as well as update the Kubernetes version and replace nodes, though in this preview those operations have manual limits documented in the project’s repository.

NICo, the Only Compatible Infrastructure Provider

NICo, the NVIDIA Infra Controller, is for now the only infrastructure provider compatible with NVKE. It’s the piece that translates NVKE’s requests into real physical machines: without NICo, there’s nowhere to provision anything.

Node Identity with SPIFFE/SPIRE

Before joining a cluster, each node authenticates through SPIFFE/SPIRE using its TPM, and only afterward receives its join credentials. Encryption of etcd data is available per cluster, as an option each workload cluster can enable separately.

sequenceDiagram
  participant N as Bare metal node
  participant T as Node TPM
  participant S as SPIFFE/SPIRE
  participant M as NVKE management plane
  N->>T: requests hardware evidence
  T-->>N: delivers attestation certificate
  N->>S: presents TPM evidence
  S-->>M: validates node identity
  M-->>N: delivers join credentials
  Note over N,M: the node joins the workload cluster

Practical Examples and Getting Started

The NVKE repository includes a local quickstart that doesn’t need an NVIDIA account or real GPU hardware: it runs on simulated machines. Before touching anything, confirm your environment meets these requirements:

• Operating system, Linux host or VM, amd64 or arm64 architecture.
• CPU and memory, at least 6 CPUs and 12 GiB of free memory.
• Disk, at least 100 GiB of free space.
• Docker, installed and running.
• Virtual TPM, virtual TPM support available on the host.

On macOS, Docker Desktop isn’t used directly: the official guide runs the evaluation inside a Colima VM. On Windows there’s no documented native support; the most direct path is spinning up a Linux VM (or WSL2 with Docker) and following the same steps as on Linux.

To explore the code before installing anything:

git clone https://github.com/dsx-ai-factory/nvke.git
cd nvke
ls cli management-engine cluster-api-provider-provisioned

This command downloads the full repository and lists three folders that are a real part of its structure: cli (the nvkectl code), management-engine (the management plane), and cluster-api-provider-provisioned (the provider that builds the workload clusters). The expected output is, literally, those three folder names.

For the full step-by-step installation, with the exact quickstart commands (which change between previews), follow the official guide linked from the project’s own README instead of copying commands from a third party.

Once the quickstart finishes creating a test workload cluster, the instructions themselves ask you to confirm its nodes are ready:

kubectl get nodes

Expected output in the STATUS column: Ready for all nodes. If any node is stuck in NotReady, the official quickstart documents that step as blocking before continuing with the rest of the evaluation.

💡 Tip: run the local quickstart first on a disposable VM. If something goes wrong, you destroy it and start over without risking any real machine.

Real-World Use Cases

• AI labs with their own GPUs, need to split that capacity into isolated clusters per project without paying a public cloud’s markup.
• GPU-as-a-service providers, can offer managed Kubernetes to their customers without building their own Cluster API provider from scratch.
• Internal platform teams, spin up a test cluster for a machine learning team in minutes instead of filing a ticket with the datacenter team.

Common Mistakes and Best Practices

The most likely mistake in this preview is confusing ‘it works’ with ‘it’s ready for production.’ NVKE leaves several documented gaps that need to be covered manually before admitting real traffic.

⚠️ Heads up: in this preview, NVKE API authentication ships turned off and traffic runs over plain HTTP after installation. Configuring it is a mandatory manual step before admitting any external call.

There’s also no automated upgrade, scaling, or replacement procedure yet for the management cluster itself, nor a backup or restore routine for it. If that cluster goes down, recovery depends entirely on whatever the platform team has planned on its own.

📌 Note: the node images with full disk encryption (FDE) that NVKE offers in this preview are for evaluation only: they don’t provide real confidentiality for data at rest.

Finally, it’s best not to mix the responsibilities of NVKE and its sibling project, NVKR (NVIDIA Kubernetes Recipes). NVKR builds NVKE’s node images and components from pinned inputs and stores the evidence of that build; NVKE only deploys those artifacts. Mixing both roles makes it harder to know, months later, exactly which version is running on each node.

NVKE vs. NVIDIA GPU Operator

NVKE and NVIDIA GPU Operator sound similar but solve different layers of the same problem.

In theory, nothing stops you from running GPU Operator inside a workload cluster created by NVKE to manage drivers at the pod level. The project’s documentation, however, doesn’t yet confirm that combination as a supported path, so it’s best treated as a hypothesis to test, not an established fact.

Going Deeper: Node Identity and the Link with NVKR

TPM attestation before handing out join credentials isn’t a cosmetic detail: it prevents a cloned or tampered machine from posing as a legitimate cluster node. SPIFFE/SPIRE gives each node a verifiable cryptographic identity, independent of its IP address or hostname, which is especially useful when nodes are physical machines that can be replaced or moved.

flowchart LR
  R["NVKR: pinned inputs"] --> I["NVKE node images and components"]
  I --> D["NVKE management plane"]
  D --> W(("Workload clusters"))

That separation between NVKR, which builds, and NVKE, which deploys, follows the same logic as separating CI from CD in a traditional software pipeline: whoever compiles and signs the artifact isn’t the one deciding, at each installation, where to run it. Even so, in a preview with no backup procedure for the management plane, that separation isn’t enough to guarantee continuity if the management cluster is lost.

Your next step: clone the repository, run the local quickstart on a disposable VM with Docker and virtual TPM, and confirm with kubectl get nodes that the test workload cluster ends up in Ready.

Frequently Asked Questions

Does NVKE Replace NVIDIA GPU Operator?

No. NVKE provisions and operates entire Kubernetes clusters on bare metal; GPU Operator installs and maintains the GPU software inside a cluster that already exists. They’re different layers of the same stack.

Can NVKE Be Used in Production Today?

NVIDIA explicitly marks it as preview: its APIs and behavior can change, and it’s not recommended for production workloads yet.

What Infrastructure Providers Does NVKE Support?

In this preview, NICo (NVIDIA Infra Controller) is the only compatible infrastructure provider.

What Does nvkectl Do Within NVKE?

It’s the CLI that assembles the management cluster using a temporary seed cluster, and that’s later used to request, scale, and delete workload clusters.

Does NVKE Run on macOS or Windows?

The official local quickstart runs natively on Linux; on macOS it runs inside a Colima VM. Windows has no documented native support, so it’s best to use a Linux VM or WSL2 with Docker.

References

• GitHub: dsx-ai-factory/nvke: official NVKE repository, with its architecture documentation, quickstart, and known limitations.
• GitHub: NVIDIA/gpu-operator: official NVIDIA GPU Operator repository, used here as a comparison reference.
• SPIFFE: official site for the SPIFFE/SPIRE project for workload and node identity.
• Cluster API (Kubernetes SIGs): official documentation for the project NVKE uses to operate workload clusters.

📱 Enjoying this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day.

Featured image: Foto de Kvistholt Photography en Unsplash