284B-Parameter MoE Model Runs on CPU Without a GPU: Here’s How

⏱️ Reading time: 14 min

A 284.33-billion-parameter MoE model on CPU running without a single GPU, on a computer with 64 GB of RAM, sounds like a typo. The llama-modchip project pulls it off with DeepSeek-V4-Flash at 44.36 tokens per second during the prefill stage and between 9.1 and 14.7 tokens per second while generating text. There’s no marketing trick behind it: three engineering techniques (aggressive quantization, selective expert offloading, and context caching) fully explain the result.

📑 En este artículo

1. TL;DR
2. What Is an MoE Model on CPU?
3. Why Running Mixture-of-Experts Without a GPU Matters
4. How Pure-RAM Inference Works
   1. Aggressive Quantization: From 82 GB to 56 GB
   2. Selective Expert Offloading: What Stays Warm and What Gets Streamed
   3. Context Caching: 256K Tokens Without Recomputing Anything
5. Practical Examples: What Performance to Expect by Machine Type
   1. How to Try It
6. Real-World Use Cases for Pure-RAM Inference
7. Common Mistakes and Best Practices
8. Comparison With Quantization Alternatives
9. Going Deeper: The Details Behind Tiling and Speculative Rollback
10. Frequently Asked Questions
    1. What is an MoE model on CPU and how does it differ from a dense one?
    2. How much RAM do I need to run DeepSeek-V4-Flash with llama-modchip?
    3. Does selective expert offloading reduce response quality?
    4. Can llama-modchip also use a GPU if I have one available?
    5. How fast does an agent’s context get restored with pure-RAM inference?
11. References

TL;DR

• DeepSeek-V4-Flash (284.33B total parameters, 13.17B active) runs entirely in 56.56 GiB thanks to quantization down to 1.71 bits per weight.
• Top-6 routing across 256 experts per layer activates barely 2.3% of the available experts at each step.
• Selective offloading keeps only the frequent experts warm and streams the cold ones without evicting the processor’s L3 cache.
• A 1.64 MB speculative drafter speeds up generation without needing a second GPU.
• Restoring a 256K-token context from /dev/shm takes between 13.1 and 32.6 milliseconds, not minutes.

What Is an MoE Model on CPU?

An MoE model on CPU is a Mixture-of-Experts system that runs full inference in the processor’s RAM, without a GPU, activating only a fraction of its total parameters per token through dynamic routing, aggressive checkpoint quantization, and caching techniques that avoid swapping data to disk.

The case that best illustrates the technique is llama-modchip, a set of patches on top of the ggml and llama.cpp kernels. It’s not a new model: it’s a quantization transcoding layer and runtime tooling that takes existing MoE checkpoints, such as DeepSeek-V4-Flash or Nemotron-3.5-30B-A3B, and adapts them to run on x86_64 processors with AVX-512 or AVX2, and on ARM64 with NEON, SVE, or Metal.

DeepSeek-V4-Flash organizes its 284.33B parameters across 43 multi-head latent attention (MLA) layers and 256 routed experts per layer. That architecture explains why quantization and offloading work layer by layer: each of the 43 MLA layers independently decides which of its 256 experts to consult for each token.

The first thing to understand is the size problem. A standard UD-IQ1_S build of DeepSeek-V4-Flash takes up 76.87 GiB (82.5 GB), while a Linux system with 64 GB of physical RAM only exposes about 62.5 GiB of usable memory. That gap, nearly 14.4 GiB, is what triggers page cache thrashing or an OOM kill before the model even finishes loading.

Why Running Mixture-of-Experts Without a GPU Matters

A mixture-of-experts model at this scale running without a GPU matters because it flips the usual relationship between model size and hardware cost. Of DeepSeek-V4-Flash’s 284.33B total parameters, only 13.17B are active in any given inference step: barely 4.6% of the total. That sparsity is what makes it feasible to run a model of this scale on a desktop machine instead of a GPU cluster.

💭 Key point: if only 4.6% of the parameters participate in each token, moving the remaining 95.4% between RAM and cache without hurting performance is the real engineering problem, not training a smaller model.

The same approach scales both up and down. On a laptop with 16 to 32 GB, a smaller model like Nemotron-3.5-30B-A3B quantized to 3.75 bpw in 14.35 GiB is enough to reach between 25.0 and 60.4 tokens per second. At the other extreme, an 8-channel Threadripper workstation with AVX-512BW saturates DDR5 bandwidth and pushes that same 30B model up to 111.62 tokens per second, also without a GPU.

Nemotron-3.5-30B-A3B takes a different path: instead of traditional attention layers, it uses 46 Mamba-2 layers, a state-space model (SSM) that doesn’t need a traditional KV cache. That lets it sustain a 256K-token context with only 432 MB of KV cache, far less than a pure attention architecture of the same size would need.

How Pure-RAM Inference Works

Pure-RAM inference rests on three design decisions operating at different layers: how much disk space the checkpoint takes up, which weights travel through the processor’s cache hierarchy at each token, and how much of the already-computed work can be reused between conversation turns.

flowchart TD
A["64 GB physical RAM"] --> B["62.5 GiB usable on Linux"]
B --> C["56.56 GiB quantized model weights"]
B --> D["KV cache at 256K context"]
B --> E["Margin for the operating system"]
subgraph Budget
C
D
E
end

Aggressive Quantization: From 82 GB to 56 GB

Quantization reduces the numerical precision of each weight so it takes up less space in RAM. A model in BF16 uses 16 bits per weight; llama-modchip pushes that figure down to an average of 1.71 bits per weight (bpw) for DeepSeek-V4-Flash, which explains why the checkpoint drops from 76.87 GiB to 56.56 GiB on disk.

Not the entire network is quantized the same way. Norms and the state of recurrent layers stay in F32 to avoid losing numerical stability, while cold experts (the ones rarely activated) get the most aggressive schemes, such as Nemotron’s Fractal-64 format, which groups 64 weights per block with a BF16 scalar and reaches 3.75 bpw without losing any score on the project’s internal quality benchmarks (100/100 on benchmarks/eval_quality).

Selective Expert Offloading: What Stays Warm and What Gets Streamed

A common, and mistaken, assumption is that llama-modchip drops experts to save on compute. By default it evaluates the six routed experts of each layer (Top-6 out of 256 total experts) with no additional pruning: the sparsity already comes from the model’s own router, not from further trimming.

What it does optimize is how those weights reach the CPU. The kernel patches replace cold-expert streaming with _MM_HINT_NTA instructions (non-temporal access), which load weights without marking them as cache-reusable data. That keeps an expert used only once per token from evicting the L3 cache lines that hold attention and the KV cache, which are reused at every step.

sequenceDiagram
participant R as Router
participant H as Warm expert
participant F as Cold expert
participant L as L3 cache
R->>H: activates frequent expert
H-->>L: reuses cache line
R->>F: activates rare expert
F-->>L: streams with non-temporal hint
Note over F,L: the cold expert does not evict the warm cache

The same patch introduces a four-level multi-column tiling scheme: AVX-512BW’s ZMM register, 32 KB L1D, 1 MB L2, and 64 MB L3, with block sizes of {8,4,3,2} columns for the quantized matrix-vector product during prefill. On a 16-core Zen 5 processor, that change pushed prefill throughput from 25.0 to 44.36 tokens per second (+77%), and a register-stationary MQA8 FlashAttention added another 53% to decode.

Context Caching: 256K Tokens Without Recomputing Anything

The third pillar is not recomputing what’s already been processed. With a 256K-token context, DeepSeek-V4-Flash’s KV cache takes up a sizable chunk of the RAM budget: the process’s peak VmRSS reaches 57.61 to 58.9 GiB, a considerable portion of which is attention cache rather than model weights.

For agent sessions that open and close context constantly, for example a 40-tool workflow in Claude Code, llama-modchip saves snapshots of the KV state in /dev/shm, RAM-backed shared memory. Restoring one of those snapshots takes between 13.1 and 32.6 milliseconds, compared to the seconds it would take to reprocess the full prompt from scratch.

To speed up token-by-token generation, the project pairs that cache with self-speculative decoding: a lightweight 1.64 MB drafter based on Zobrist n-grams (adapted from a chess algorithm, Stockfish-PVS v2) proposes several candidate tokens that the large model verifies in parallel instead of generating them one at a time. An $O(1)$-time speculative rollback fix prevents the recurrent and MLA states of deepseek4-type architectures from getting corrupted when the drafter guesses wrong and a token has to be discarded.

flowchart LR
A["Stockfish-PVS drafter 1.64 MB"] --> B["Proposes candidate tokens"]
B --> C["Large model verifies in parallel"]
C -->|"accepted tokens"| D["Added to output"]
C -->|"rejected token"| E["O(1) rollback of MLA states"]
E --> A

Practical Examples: What Performance to Expect by Machine Type

The performance of a GPU-free MoE depends far more on the hardware class than on the model’s nominal size. This table summarizes the four configurations the project itself documents:

How to Try It

The repository includes an install.sh script at the root and specific guides for Apple Silicon, Fedora, and Windows in its apple-silicon/, fedora/, and windows/ folders. Since the exact dependencies (compiler version, AVX-512 flags, llama.cpp submodules) vary by distribution and CPU, the safest route is to follow the quickstart in the official README instead of copying a generic command: a wrong compiler flag for your processor can keep the binary from starting at all.

Real-World Use Cases for Pure-RAM Inference

The most concrete case the project documents is coding agents that open long sessions. A typical Claude Code workflow with 40 active tools directly benefits from restoring context from /dev/shm in milliseconds instead of reprocessing the full prompt on every session restart.

Another use case is developers who want to evaluate LLMs on CPU with full MoE architectures, complete with their 256 experts and real router, without paying for cloud GPUs. And at the hybrid-hardware end, a 16 GB GPU is enough to keep the attention backbone warm while cold experts stay in system RAM, a combination that in the project’s benchmark reaches 80.80 tokens per second with Nemotron-30B or Qwen3-30B.

Common Mistakes and Best Practices

The most frequent mistake is assuming that any public build of an MoE model fits in 64 GB. A standard UD-IQ1_S build of DeepSeek-V4-Flash weighs 76.87 GiB: you specifically need llama-modchip’s 56.56 GiB format for it to fit the real budget.

⚠️ Watch out: always calculate against the 62.5 GiB usable on a 64 GB Linux system, not against the nominal 64 GB: the operating system and buffers claim the rest.

Another mistake is confusing selective offloading with expert pruning: as already explained, the project evaluates all six experts of each layer without trimming any. It optimizes data movement, not the amount of compute.

The third trap is ignoring swap. The entire design targets zero NVMe swapping: if the system starts paging to disk, latency spikes even though the process technically keeps running. To confirm which quantization format actually loaded, the binary’s boot log prints the effective bpw of each tensor; if that detail doesn’t show up in your build, it’s a sign you’re running the standard checkpoint, not the llama-modchip one. And since performance depends on the available SIMD instruction set, it’s worth benchmarking on your own CPU before assuming the 44.36 tok/s from the 16-core Zen 5 will carry over to a processor with only AVX2.

Comparison With Quantization Alternatives

Not every quantization format makes the same tradeoff between size and quality. The following table compares the three variants llama-modchip offers for Nemotron-3.5-30B-A3B:

Using a full GPU is still faster in absolute terms, but it requires video memory that a 284B model simply can’t fit on a consumer card. Pure-RAM inference doesn’t compete on peak speed: it competes on the fact that it runs at all, with no extra hardware investment.

Going Deeper: The Details Behind Tiling and Speculative Rollback

The four-level tiling scheme isn’t just a cache optimization: it’s a change in the order in which the weight matrix’s columns are traversed. Processing several columns at once per register block reduces how many times the same L2 or L3 block has to be fetched again from RAM, which is the most expensive operation on a modern processor.

The tiling block isn’t a fixed number: the runtime evaluates the {8,4,3,2} column combinations per cache level and picks whichever gives the best performance for each layer’s matrix size, since a layer with more active experts per token doesn’t benefit the same way from an 8-column block as a layer with fewer.

The $O(1)$-time speculative rollback solves a problem specific to architectures with recurrent states like Mamba-2 or MLA: unlike a pure transformer, where discarding a rejected speculative token is trivial, these models accumulate a compressed state that changes with every processed token. Without an efficient rollback mechanism, undoing a badly speculated token would require recomputing the full state from the last valid checkpoint, which would wipe out the gains from speculative decoding.

Your next step: clone the llama-modchip repository and run the Nemotron-3.5-30B-A3B model in Q4_0 format first to measure how many tokens per second your own CPU delivers, before trying to load the full 284B checkpoint.

Frequently Asked Questions

What is an MoE model on CPU and how does it differ from a dense one?

An MoE model on CPU activates only a fraction of its parameters per token through a router, while a dense model uses 100% of its weights at every step. That sparsity is what lets a model with 284B total parameters only need to move 13.17B active parameters per token.

How much RAM do I need to run DeepSeek-V4-Flash with llama-modchip?

The project reports a peak VmRSS of 57.61 to 58.9 GiB with a 256K-token context, within the 62.5 GiB usable on a Linux system with 64 GB of physical RAM. With less than that, the system falls back to swap or fails outright with an OOM.

Does selective expert offloading reduce response quality?

No. By default, the six routed experts of each layer are evaluated (Top-6 out of 256), just as in an unoptimized build. What changes is how those weights travel through the cache hierarchy, not how many experts participate in the computation.

Can llama-modchip also use a GPU if I have one available?

Yes, in hybrid mode: it keeps the attention backbone and SSM layers on the GPU with -ngl 99 while leaving the cold MoE experts in system RAM. In that configuration, with a 16 GB GPU, the project reports 80.80 tokens per second with Nemotron-30B or Qwen3-30B.

How fast does an agent’s context get restored with pure-RAM inference?

Between 13.1 and 32.6 milliseconds for a KV cache snapshot saved in /dev/shm, depending on prompt size. The documented case is a 40-tool Claude Code workflow, well below the seconds it would take to reprocess the prompt from scratch.

References

• GitHub: synchopate/llama-modchip: official repository with the README, benchmarks, and architecture documentation for the project.
• GitHub: ggml-org/llama.cpp: the inference runtime on which llama-modchip applies its kernel patches.
• Wikipedia: Mixture of experts: general explanation of the conditional routing architecture these models use.
• GitHub: DeepSeek AI: official organization that publishes the base checkpoints for the DeepSeek-V4 family.

📱 Enjoying this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day. @programacion

Featured image: Foto de Brian Kostiuk en Unsplash