Local LLM Speed: What Determines Tokens per Second

⏱️ Reading time: 14 min

Buying a high-end GPU and discovering that a heavy model runs much slower than expected is an expensive surprise. The llmspeed.dev simulator aims to solve exactly that: letting you feel, before paying for the hardware, what determines local LLM speed depending on the GPU, unified memory, or CPU you’re considering.

📑 En este artículo

1. TL;DR
2. What is local LLM speed?
3. Why local LLM performance matters
4. How local inference for language models works
   1. The memory bandwidth bottleneck
   2. Quantization: fewer bytes per parameter
   3. Dense vs MoE: how many parameters activate per token
   4. Prefill vs decode
   5. The memory cost of context
5. Practical examples and how to get started
   1. How to run the simulator on your own machine
6. Real-world use cases
7. Common mistakes and best practices
8. Comparing methods for estimating local LLM performance
9. Going deeper: the roofline model applied to inference
10. Frequently Asked Questions
    1. What exactly does local LLM speed measure?
    2. Why doesn’t a Mac with 512GB always give more local inference speed than a GPU with 24GB?
    3. What does the “A3B” suffix mean in an MoE model and how does it affect local tokens per second?
    4. Does quantization hurt quality while speeding up local inference for language models?
    5. Where can I compare local LLM performance before buying hardware?
    6. What matters more for local LLM performance: TFLOPS or memory bandwidth?
11. References

The project is open source on GitHub and cross-references dozens of GPUs, Apple Silicon machines, and CPU-only configurations against a list of popular open models. The number that shows up on screen depends on a handful of technical factors that are almost never explained together, and those are exactly what this article covers.

TL;DR

• Memory bandwidth, not TFLOPS, is the real limit on local LLM speed when generating tokens.
• You calculate tokens per second with a simple formula: bandwidth divided by the total bytes of the active model.
• MoE models with an “A3B” suffix activate only a fraction of their total parameters per token.
• 4-bit quantization cuts bytes per parameter in half compared to 8-bit and speeds up inference.
• Llmspeed.dev combines benchmarks from r/LocalLLaMA and localscore.ai to estimate speed before buying hardware.

What is local LLM speed?

Local LLM speed is the number of tokens a language model generates per second when running on the user’s own hardware, whether a GPU, a Mac, or a CPU, instead of a remote server. It depends on memory bandwidth, the size of the active model, and quantization.

This speed is not a fixed property of the model: the same weights file can produce 10 tokens per second on one machine and 10 times more on another, depending on how those three factors relate to the specific hardware.

Why local LLM performance matters

A model that runs on your own machine doesn’t depend on an external API, doesn’t send your data to a third party, and has no per-token cost. But if inference is too slow, those advantages become theoretical: nobody waits several seconds between each word from a coding assistant.

The problem is that a GPU’s or Mac’s specs don’t directly tell you how many tokens per second you’ll get with a specific model. A card with more VRAM isn’t always faster: it can have more capacity but less bandwidth than another card with less memory. That’s why tools like llmspeed try to translate raw specs into a concrete sense of speed before you spend the money.

How local inference for language models works

The memory bandwidth bottleneck

Generating a new token requires reading, once again, every active parameter of the model from memory into the compute cores. That trip repeats at every generation step, so the speed ceiling isn’t set by raw compute power but by how many bytes per second the memory bus can move.

That’s the same reason a dedicated GPU with GDDR6X memory usually beats a Mac with unified memory in raw tokens per second, even though the Mac has far more capacity. Capacity decides which model fits in memory; bandwidth decides how fast it runs once it’s there.

flowchart LR
    A["Model weights in VRAM"] --> B["Memory bus (GB/s)"]
    B --> C["GPU compute cores"]
    C --> D["Generated token"]
    D --> A

Quantization: fewer bytes per parameter

Quantizing a model means storing each parameter with fewer bits: from 16 bits (FP16) down to 8 or 4 bits, for example. Fewer bits per parameter means fewer bytes to move across the memory bus for each token, so quantization speeds up local inference without changing a single line of the model.

The trade-off is precision. A model quantized to 4 bits loses some quality compared to the same model in FP16, although in practice the difference tends to be smaller than expected for large models. The llmspeed repository lets you choose the quantization level when simulating each model and hardware combination.

Dense vs MoE: how many parameters activate per token

A dense model uses all of its parameters for every token it generates. A Mixture of Experts (MoE) model, on the other hand, has a router that picks, for each token, only a few experts out of many available. The Qwen3 30B A3B model on llmspeed’s list illustrates the naming convention: 30B is the total parameter count, and the A3B suffix means only 3 billion activate per token.

That difference matters a lot for speed. A 30B MoE with 3B active parameters behaves, in bytes read per token, much more like a dense 3B model than a dense 30B model. That’s why models like GPT-OSS 120B can feel faster than their total size suggests.

flowchart TD
    A["Total model parameters"] --> B{"Architecture"}
    B -->|"Dense"| C["All parameters active on every token"]
    B -->|"MoE"| D["Router selects a few experts per token"]
    D --> E["Only a fraction of parameters active"]

Prefill vs decode

Processing a long prompt and generating the response aren’t the same task for the hardware. The prefill phase (reading the full prompt) is parallel-compute intensive and tends to be fast on powerful GPUs. The decode phase (generating each new token, one at a time) is the one that suffers from the memory bottleneck described above.

sequenceDiagram
    participant U as User
    participant M as Local model
    U->>M: sends the full prompt
    M->>M: prefill phase, processes the entire prompt in parallel
    M-->>U: first token
    loop for each new token
        M->>M: decode phase, reads the active parameters
        M-->>U: next token
    end

The memory cost of context

As a conversation grows, the model keeps a key-value cache (KV-cache) with information from every previous token so it doesn’t have to recompute it. That cache also lives in memory and competes for the same bandwidth as the model’s parameters.

That’s why the speed measured at the start of a short conversation doesn’t hold steady in a long one: the more the context grows, the more memory the KV-cache occupies, and the less bandwidth remains available to read the parameters at each decoding step.

Practical examples and how to get started

Before simulating anything, it helps to understand the math the simulator does under the hood. The simplest formula for estimating tokens per second in the decode phase is to divide the available memory bandwidth by the total bytes that need to be read for each token.

def tokens_por_segundo(bandwidth_gbps, params_billion, bytes_por_param=2):
    bytes_totales = params_billion * 1e9 * bytes_por_param
    return (bandwidth_gbps * 1e9) / bytes_totales

# Illustrative values, they don't correspond to a real GPU:
print(tokens_por_segundo(bandwidth_gbps=500, params_billion=8))

This prints 31.25. With a hypothetical bandwidth of 500 GB/s and a dense model of 8 billion parameters in FP16 (2 bytes per parameter), the theoretical ceiling for the decode phase would be just over 31 tokens per second.

The second step is to account for how many parameters actually activate. MoE models indicate that figure in their own name:

def parametros_activos(nombre_modelo):
    # The "AxB" convention indicates how many billions activate per token
    sufijo = nombre_modelo.split()[-1]
    if sufijo.startswith("A") and sufijo.endswith("B"):
        return float(sufijo[1:-1])
    return None

print(parametros_activos("Qwen3 30B A3B"))

The result is 3.0. Despite its 30 billion total parameters, Qwen3 30B A3B only activates 3 billion per token, the same volume a much smaller dense model would move.

Combining quantization and MoE architecture, the math changes radically:

def tokens_por_segundo_moe(bandwidth_gbps, activos_billion, bits_por_param=4):
    bytes_por_param = bits_por_param / 8
    bytes_totales = activos_billion * 1e9 * bytes_por_param
    return (bandwidth_gbps * 1e9) / bytes_totales

# Qwen3 30B A3B quantized to 4 bits, same hypothetical bandwidth of 500 GB/s:
print(tokens_por_segundo_moe(bandwidth_gbps=500, activos_billion=3))

The result jumps to 333.33 theoretical tokens per second, ten times more than the dense model in the first example, even though both run on the same hypothetical bandwidth. This is exactly the kind of comparison llmspeed automates for dozens of real model and hardware combinations.

How to run the simulator on your own machine

Llmspeed is, according to its own repository, open source. To run it locally you only need git and a static file server, and Python already ships with one built in:

git clone https://github.com/nicorobinreal/llmspeed.git
cd llmspeed
python3 -m http.server 8000

The terminal responds with something like:

Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...

Then open http://localhost:8000 in your browser and you can already choose hardware and model from the interface. On Windows, the equivalent command in PowerShell or CMD is python -m http.server 8000, because there the executable is called python instead of python3.

Real-world use cases

A developer coding with locally running assistants needs the decode phase to be fast: every multi-second pause between tokens breaks the workflow. For that case, what matters is the bandwidth available to the chosen model, not the total amount of free VRAM.

A research team that needs to load a huge model to experiment with, even if slowly, prioritizes memory capacity over raw speed. That’s where machines with lots of unified memory come in, like Mac Studio M5 Ultra 512GB and Mac Studio M3 Ultra 512GB, which can load models that no consumer GPU could even fit in VRAM.

A tight budget that can only afford a CPU and system RAM, with no dedicated GPU, is the third scenario. Llmspeed explicitly lists a CPU-only (Ryzen 9 9950X, 96GB DDR5) configuration for that case, where the realistic speed expectation is much lower than for any GPU on the list.

Common mistakes and best practices

• Confusing capacity with speed: having 512GB of unified memory doesn’t mean high tokens per second. Capacity decides which model fits; bandwidth decides how fast it runs.
• Ignoring context growth: as a conversation gets longer, the KV-cache occupies more memory and competes for the same bandwidth, so real speed drops in long conversations.
• Looking only at the first token: the prefill phase can feel instant on a powerful GPU while the decode phase, the one that determines the real usage experience, remains slow.
• Assuming every number is measured: llmspeed mixes real benchmarks from communities like r/LocalLLaMA and localscore.ai with estimates projected from specs, for combinations nobody has tested yet.
• Comparing different quantizations without noting it: a model at 4 bits and the same model at 16 bits aren’t comparable in speed or quality without first fixing the quantization level.

Comparing methods for estimating local LLM performance

Going deeper: the roofline model applied to inference

The formula of bandwidth divided by bytes is a simplification of the roofline model, a classic computer architecture tool for predicting whether a workload is memory-bound or compute-bound. For an LLM’s decode phase, arithmetic intensity (compute operations per byte read) is low, so the ceiling is almost always set by memory, not the GPU.

That changes with batch size: if several requests share the same parameter-reading pass, the memory cost gets spread across more tokens generated at once, pushing the workload closer to the compute limit. That’s why a server handling hundreds of simultaneous users can be far more efficient per token than a single local user generating one at a time.

Speculative decoding, which uses a small model to propose several tokens and a large one to verify them in a single pass, attacks the same bottleneck from another angle. Instead of speeding up the memory read, it reduces how many times it needs to be repeated.

💭 Key takeaway: in local inference it’s almost always better to optimize for memory bandwidth before TFLOPS, because that’s the resource that genuinely runs out first.

Your next step: clone the llmspeed repository, run python3 -m http.server 8000, and compare two pieces of hardware from your shopping list with the same model to see how much the estimated speed changes.

Frequently Asked Questions

What exactly does local LLM speed measure?

It measures how many tokens per second a language model generates while running on the user’s own hardware, separating the prefill phase (processing the prompt) from the decode phase (generating each new token).

Why doesn’t a Mac with 512GB always give more local inference speed than a GPU with 24GB?

Because memory capacity and bandwidth are different things. A Mac’s unified memory lets you load huge models, but it tends to have less bandwidth than a GPU’s dedicated GDDR6X memory, so the GPU can win on tokens per second with models that fit within its 24GB.

What does the “A3B” suffix mean in an MoE model and how does it affect local tokens per second?

It indicates that, out of the 30 billion total parameters in a model like Qwen3 30B A3B, only 3 billion activate to generate each token. Fewer active parameters means fewer bytes to read per token, so these models tend to deliver more tokens per second locally than a dense model of the same total size.

Does quantization hurt quality while speeding up local inference for language models?

It reduces the numerical precision of each parameter, which can affect quality, but in large models the difference between 8 bits and 4 bits tends to be smaller than people expect. The speed gain, on the other hand, is direct: fewer bytes per parameter means less read time at each token.

Where can I compare local LLM performance before buying hardware?

Llmspeed.dev gathers a list of GPUs, Apple Silicon machines, and CPU-only configurations cross-referenced with popular open models, combining benchmarks from communities like r/LocalLLaMA with spec-based projections.

What matters more for local LLM performance: TFLOPS or memory bandwidth?

For the decode phase, almost always memory bandwidth. TFLOPS matter more during prefill, when the model processes the entire prompt in parallel before it starts generating.

References

• llmspeed.dev: the original simulator this article is based on, with benchmarks by hardware and by model.
• github.com/nicorobinreal/llmspeed: open source code for the simulator and the hardware and model data it uses.
• localscore.ai: a standardized local inference benchmark that llmspeed uses as one of its measured data sources.
• r/LocalLLaMA: a Reddit community that shares real benchmarks of models running on consumer hardware.

📱 Like this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day.

Featured image: Foto de Maxence Pira en Unsplash