Kimi Delta Attention: The Linear Attention Behind Kimi K2 and Qwen3

⏱️ Lectura: 13 min

Kimi K2 and the recent models in the Qwen3 family no longer compare each token against all previous ones at every attention step: they use Kimi Delta Attention (KDA), a mechanism that stores all the relevant history in a fixed-size state and updates it token by token. It’s the architectural piece that lets these models process long contexts without the computational cost growing quadratically with sequence length.

📑 En este artículo

1. TL;DR
2. What happened
3. From quadratic attention to fixed state: a brief history
4. How Kimi Delta Attention works step by step
5. DeltaNet, Gated DeltaNet, and KDA: comparison table
6. How to try it today
7. Impact and analysis
8. What’s next
9. Frequently Asked Questions
   1. What does it mean that linear attention has linear cost?
   2. Why does DeltaNet need keys to be normalized?
   3. Does KDA completely replace softmax attention in Kimi and Qwen3?
   4. What’s the concrete difference between Gated DeltaNet and KDA?
   5. Can I train a model from scratch using these layers?
   6. Where can I find the code or the weights of the models that use KDA?
10. References

A technical article published on the Doubleword blog, ‘You Could Have Come Up With Kimi Delta Attention’, reconstructs the full derivation: it starts from classic softmax attention, removes the normalization, arrives at linear attention, then DeltaNet, then Gated DeltaNet, and finally KDA. The result is a chain of design decisions that, seen one by one, stop looking like magic.

TL;DR

• Kimi Delta Attention (KDA) is the linear attention mechanism used by Moonshot AI’s Kimi models and the Qwen3 family.
• KDA is the latest installment in a family that starts with DeltaNet and passes through Gated DeltaNet.
• Unlike softmax, which compares each token against all previous ones (quadratic cost), KDA stores history in a fixed-size state (linear cost).
• KDA’s key equation applies a per-channel decay Diag(α_t) to the previous state and corrects the error with a delta rule scaled by β_t.
• The reference code for DeltaNet, Gated DeltaNet, and other linear attention variants is in the flash-linear-attention repository on GitHub.
• Moonshot AI publishes the weights of its Kimi models on Hugging Face under the moonshotai organization.
• The article documenting this step-by-step derivation is ‘You Could Have Come Up With Kimi Delta Attention’, from the Doubleword blog.

What happened

Doubleword’s technical blog published a didactic derivation of Kimi Delta Attention, the attention mechanism Moonshot AI uses in its Kimi family and which, according to the article itself, was also adopted by recent Qwen models. Instead of presenting KDA’s equations as a closed block, the author reconstructs them from scratch, showing what problem each term solves.

This matters because modern linear attention variants (RetNet, RWKV, Mamba2, DeltaNet, Gated DeltaNet, KDA) tend to be presented with dense notation that hides the core idea. Without that context, a technical reader sees an equation with diagonal matrices and outer products and doesn’t understand why each piece exists.

From quadratic attention to fixed state: a brief history

Softmax attention calculates, for each token, a similarity score against all previous tokens, normalizes those scores with an exponential, and uses the result to average the value vectors. It’s precise, but the cost grows with the square of the sequence length: with T tokens there are T² key-query pairs.

Linear attention comes from a simple observation: if the softmax normalization is removed, the scalar dot product between key and query can be moved around inside the sum. This allows grouping everything that depends on the past into a single fixed-size matrix, the state S, instead of storing each key and value separately.

The identity that makes this trick possible is this: an outer product |v⟩⟨k| is a matrix, and applying it to a query |q⟩ gives ⟨k|q⟩|v⟩, exactly the same as first calculating the dot product ⟨k|q⟩ and then scaling the vector v. Summing these outer products token by token incrementally builds the same computation that previously required looking at the entire history again at each step.

DeltaNet fixes a limitation of pure linear attention: if only outer products are summed, the state never “forgets” or corrects old information, and with long sequences it can saturate. DeltaNet introduces a delta rule: instead of simply adding the new value, it first predicts what value the current state would “remember” for that key, calculates the error between the real value and that prediction, and writes only that error. Gated DeltaNet additionally adds a decay (gate) that attenuates the previous state before applying the delta correction, giving the model an explicit way to forget irrelevant information over time.

How Kimi Delta Attention works step by step

Kimi Delta Attention takes this idea one step further: instead of a single (scalar) decay for the whole state, it applies a different decay per channel via a diagonal matrix Diag(α_t). This gives the model fine control over which state dimensions attenuate faster and which are preserved longer, something a scalar decay can’t express.

The full sequence of operations KDA runs at each token t is as follows: first it decays the previous state channel by channel, then it uses that decayed state to predict what value it remembers for the current key, calculates the error between the real value and that prediction scaled by a factor β_t, corrects the state with that error, and finally reads the output by projecting the corrected state onto the current query (scaled by the inverse square root of the key dimension, just like in softmax).

Translated into code, the simplest version of linear attention (without the delta rule, just to understand the state read/write mechanics) looks like this:

import torch

def linear_attention_step(S, k_t, v_t, q_t):
    # S: state of shape (d_v, d_k), accumulates the history
    S = S + torch.outer(v_t, k_t)   # write: add v_t x k_t
    o_t = S @ q_t                   # read: project the query onto the state
    return S, o_t

This version only accumulates, it never corrects. The equivalent step for Kimi Delta Attention, with per-channel decay and the delta rule, adds three more operations:

import torch

def kda_step(S, k_t, v_t, q_t, alpha_t, beta_t, d_k):
    # alpha_t: per-channel decay vector (Diag(alpha_t))
    S_tilde = S * alpha_t.unsqueeze(0)   # decay the previous state, channel by channel
    v_hat_t = S_tilde @ k_t              # what value the state "remembers" for this key
    e_t = beta_t * (v_t - v_hat_t)       # error between the real value and the remembered one
    S = S_tilde + torch.outer(e_t, k_t)  # correct the state with the delta rule
    o_t = S @ (q_t / d_k ** 0.5)         # read the state with the scaled query
    return S, o_t

Each call to kda_step processes one token and returns the updated state along with the output for that position. In production this doesn’t run as a Python loop token by token (it would be too slow): it runs with Triton kernels that process the sequence in blocks (chunkwise), but the math logic is exactly what these five lines show.

The following diagram summarizes the read/write cycle KDA runs at each step:

flowchart TD
    A["Token t: k_t, v_t, q_t"] --> B["Decay the state: Stilde = Sprev x Diag(alpha_t)"]
    B --> C["Predict: v_hat = Stilde x k_t"]
    C --> D["Calculate error: e_t = beta_t x (v_t - v_hat)"]
    D --> E["Correct: S_t = Stilde + e_t x k_t"]
    E --> F["Read output: o_t = S_t x q_t"]
    F --> G["S_t passes to token t+1"]

💭 Key point: the identity |v⟩⟨k|q⟩ = ⟨k|q⟩|v⟩ is the only mathematical step that turns a sum of T² comparisons into a fixed-size state update per token. Everything else (DeltaNet, Gated DeltaNet, KDA) is an increasingly fine-grained way of deciding what gets written into that state.

DeltaNet, Gated DeltaNet, and KDA: comparison table

Each variant in this family adds a mechanism on top of the previous one. The following table summarizes what each one contributes and its resulting per-token cost, compared against the original softmax attention:

Variant	What it adds	Cost per token	Where it appears
Softmax attention	Normalizes and compares each token against the entire history	O(T) per token, O(T²) total	Classic Transformer
Linear attention (no softmax)	Collapses history into a fixed-size state	O(1) per token	Theoretical basis for variants like RetNet
DeltaNet	Delta rule: corrects the state instead of just accumulating	O(1) per token	DeltaNet
Gated DeltaNet	Scalar decay (gate) before the delta correction	O(1) per token	Recent hybrid architectures
Kimi Delta Attention (KDA)	Per-channel decay Diag(α_t) + delta rule with β_t	O(1) per token	Kimi (Moonshot AI), Qwen3

How to try it today

The reference implementation for DeltaNet, Gated DeltaNet, and other linear attention variants lives in the flash-linear-attention repository on GitHub, with kernels written in Triton. It requires an NVIDIA GPU to run at real speed; on CPU or Apple Silicon it runs in eager mode (slower, useful only for understanding the logic).

Installation on Linux:

python3 -m venv fla-env
source fla-env/bin/activate
pip install --upgrade pip
pip install flash-linear-attention transformers torch

Installation on macOS (Apple Silicon, no Triton kernels, runs in eager mode):

python3 -m venv fla-env
source fla-env/bin/activate
pip install --upgrade pip
pip install flash-linear-attention transformers torch

Installation on Windows (Triton kernels need WSL2 with an NVIDIA GPU):

wsl --install -d Ubuntu
wsl
# inside WSL2, repeat the Linux installation steps

Once installed, a DeltaNet layer can be instantiated and tested with a sample tensor:

from fla.layers import DeltaNet
import torch

layer = DeltaNet(hidden_size=1024, num_heads=8)
x = torch.randn(2, 128, 1024)  # (batch, length, hidden)
out, *_ = layer(x)
print(out.shape)  # torch.Size([2, 128, 1024])

To confirm that the state stays fixed-size (and doesn’t grow with sequence length like in softmax), the most direct way is to measure peak memory with different input lengths and compare:

torch.cuda.reset_peak_memory_stats()
out, *_ = layer(x)
print(torch.cuda.max_memory_allocated() / 1e6, "MB")

If this measurement is repeated while doubling the sequence length and peak memory stays practically constant, that’s direct evidence the layer is operating in fixed-state recurrent mode and not recomputing full attention.

💡 Tip: before writing your own Triton kernels to experiment with linear attention variants, it’s worth starting from the flash-linear-attention code: it already handles chunkwise mode (processing the sequence in blocks), which is what makes training these models at real scale feasible.

Impact and analysis

The reason this family of mechanisms matters in production is the cost of serving long context. In standard softmax attention, the key-value cache (KV cache) grows proportionally to the length of the conversation, and each new token has to compare against that entire cache. In a fixed-state mechanism like KDA, the “cache” is the state matrix S, whose size doesn’t depend on how many tokens have already been processed.

The fact that Moonshot AI and models in the Qwen3 family have adopted variants of this research line, as documented in the Doubleword article, signals that the problem of softmax’s quadratic cost in long contexts is no longer solved just with more GPU memory: it’s also solved by changing the attention architecture.

⚠️ Watch out: DeltaNet’s derivation assumes keys arrive normalized. If that normalization isn’t applied correctly in the implementation, the delta rule can become numerically unstable and the state diverges in long sequences.

The trade-off paid for this saving is the loss of the exact selectivity that softmax provides: exponential normalization lets a model “ignore” irrelevant tokens almost completely and very precisely, while a fixed-size state, no matter how fine its gating, compresses information and can lose detail in extremely long sequences. That’s why several hybrid architectures combine linear attention layers with interleaved full attention layers, instead of replacing softmax throughout the network.

What’s next

The Doubleword article itself frames the derivation as a foundation for understanding future variants: the DeltaNet, Gated DeltaNet, KDA family isn’t an endpoint but a progression, and each new generation of models that needs longer contexts with lower inference cost is a candidate for introducing another form of gating or correction on the same skeleton (decay, predict, correct, read).

For a team evaluating linear attention architectures today, the practical starting point is still the same: read the implementation in flash-linear-attention, run the repository’s tests, and compare memory and numerical stability against a standard softmax attention layer on your own use case, before deciding whether the trade-off is worth it.

📖 Summary on Telegram: View summary

Try it yourself: clone flash-linear-attention, install the dependencies with the block above, and run the DeltaNet snippet to see the fixed-size state in action in minutes.

Frequently Asked Questions

What does it mean that linear attention has linear cost?

It means the work per token doesn’t grow with the number of tokens already processed. In softmax, token 10,000 compares against the previous 9,999; in linear attention, it compares against a fixed-size state that already summarizes that history.

Why does DeltaNet need keys to be normalized?

The delta rule calculates an error between the real value and the value the state “predicts” for a given key. If the keys aren’t normalized, that prediction loses the correct scale, and the correction can destabilize the state in long sequences.

Does KDA completely replace softmax attention in Kimi and Qwen3?

The Doubleword article describes the mathematical derivation of KDA as a linear attention mechanism; many architectures that adopt this type of mechanism combine it with interleaved full attention layers instead of eliminating softmax from the entire network.

What’s the concrete difference between Gated DeltaNet and KDA?

Gated DeltaNet applies a scalar decay (a single number) to the whole state before the delta correction. KDA applies a per-channel decay via the diagonal matrix Diag(α_t), giving finer control over which state dimensions get forgotten faster.

Can I train a model from scratch using these layers?

The flash-linear-attention repository exposes layers like DeltaNet ready to integrate into a standard transformer architecture, replacing the attention block; training a complete model also requires the rest of the training pipeline (data, tokenizer, optimization loop).

Where can I find the code or the weights of the models that use KDA?

The linear attention layer code is in flash-linear-attention on GitHub. The weights of Moonshot AI’s Kimi models are published on Hugging Face under the moonshotai organization.

References

• You Could Have Come Up With Kimi Delta Attention: the original Doubleword article that derives KDA step by step from softmax attention.
• flash-linear-attention: GitHub repository with Triton kernels for DeltaNet, Gated DeltaNet, and other linear attention variants.
• moonshotai on Hugging Face: organization where Moonshot AI publishes the weights of its Kimi models.
• Attention (machine learning): general context on the attention mechanism in neural networks, on Wikipedia.

📱 Like this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day.

Imagen destacada: Foto de Logan Voss en Unsplash