Gated DeltaNet: What It Is and How It Works in Qwen3.5

⏱️ Reading time: 12 min

A 4-billion-parameter language model runs a complete RISC-V processor without any external plugin or sandbox. That’s how Qwenputer works, the experiment by researcher Divyansh Agrawal on Qwen3.5. The piece that makes it possible is Gated DeltaNet, the linear attention layer the original model already uses, reconfigured to work as exact, addressable RAM memory.

📑 En este artículo

1. TL;DR
2. What Is Gated DeltaNet?
3. Why This Addressable Memory Matters
4. How the Delta Rule Works Under the Hood
5. Practical Examples: Simulating the Memory in a Few Lines of Python
6. Real Use Cases for This Linear Memory
7. Common Mistakes and Best Practices
8. Comparison: Standard Attention, Gated Linear Attention, and Other Alternatives
9. Going Deeper: Limits and Advanced Math
10. Frequently Asked Questions
    1. What is Qwenputer?
    2. Does Qwen3.5 use this exact memory by default?
    3. What is the delta rule?
    4. Why doesn’t this memory’s state grow with the conversation?
    5. Can real code actually run inside Qwen3.5’s weights?
11. References

The experiment is a curiosity, but the layer it exploits is real and runs in production: Alibaba uses it in Qwen3-Next and Qwen3.5. That linear attention layer explains how these models handle long contexts without the per-token cost growing with every new message.

TL;DR

• Fixing the gate at 1.0 and using one-hot keys turns the linear attention state into an exact row of RAM.
• Qwenputer compiles a RISC-V RV32IM CPU into 11 of Qwen3.5’s 32 layers, with no additional training.
• The state of each linear attention head stays the same size no matter how many tokens the conversation has, unlike the KV-cache.
• By default, Qwen3.5 uses a gate below 1.0, so memory fades instead of being written permanently.
• The delta rule from Schlag, Irie, and Schmidhuber (2021) already described this associative memory; Qwenputer makes it exact and addressable.

What Is Gated DeltaNet?

Gated DeltaNet is a linear attention layer that Qwen3-Next and Qwen3.5 use in 3 out of every 4 attention blocks, instead of scanning the entire history like standard attention does. It keeps a state matrix per head that updates token by token with a forget gate and a write rule, the delta rule.

The idea isn’t new. In 2021, Schlag, Irie, and Schmidhuber showed that linear attention with the delta rule works as an associative memory, or “fast weights”. Qwenputer takes that same math and pushes it to an extreme: gate at 1.0, write strength at 1.0, and keys encoded as a single 1 among zeros (one-hot). The result stops being a fuzzy memory and becomes addressable RAM, row by row.

Why This Addressable Memory Matters

Standard transformer attention stores every token in a key-value cache that grows without limit. The longer the conversation, the more memory and compute each new step needs. Gated linear attention solves that problem with a different trick: it compresses the entire past into a fixed-size matrix per head, so the per-token cost doesn’t rise with context length.

The problem is that, in its standard form, that compression is lossy. The default gate mixes old and new values, so there’s no way to read an exact value back. Qwenputer shows that if the gate and write strength are pushed to the right extremes, the same layer can store exact values without losing the fixed-size advantage. That’s what makes it possible to simulate registers, RAM, and even a rudimentary disk inside the model’s weights.

For teams running these models in production, the fixed state size matters for a practical reason: the memory each active conversation takes up on a server doesn’t depend on how long the user has been talking. That simplifies capacity decisions that, with standard attention and its growing KV-cache, become harder to predict as conversations get longer.

How the Delta Rule Works Under the Hood

Each gated linear attention head keeps a state: a matrix that works as its memory. At every token, before writing anything new, the state gets multiplied by the gate, a number between 0 and 1. A gate of 0.9 slightly erases everything stored at every step; a gate of 1.0 erases nothing.

After the gate, the head nudges one direction of the state toward a new value, and the write strength (beta) decides by how much. With beta at 1.0, the old value in that direction gets fully replaced instead of blended. And if the key marking that direction is one-hot (a vector of zeros with a single 1), the write lands exactly on a single row of the matrix, just like a memory address points to a single RAM cell.

The trick isn’t training a new model, it’s fixing three numbers that already existed in the architecture: the gate, the write strength, and the shape of the key. Not a single extra parameter is needed.

flowchart TD
A["Token generated by Qwen3.5"] --> B["One-hot key: a single 1 among zeros"]
B --> C["Gate set to 1.0"]
C --> D["Write strength beta = 1.0"]
D --> E[("Row of the state matrix")]
E --> F["Exact write: overwrites that row"]

This diagram summarizes the path of a single write. Reading works the same way, but in reverse: a one-hot query selects that same row and returns it without blending it with any other.

graph LR
A["Standard attention"] --> B["KV-cache grows with every token"]
C["Gated linear attention"] --> D["Fixed-size state per head"]

The core difference is this: standard attention never forgets anything because it keeps the entire history, while gated linear attention actively decides what to keep. Pushing that decision to the extremes is what turns controlled forgetting into exact memory.

Practical Examples: Simulating the Memory in a Few Lines of Python

You don’t need Qwen3.5’s weights to see why the trick works. These two examples reproduce the behavior of a single linear attention head using numpy, with no deep learning framework at all.

The first example shows the default behavior: a gate below 1.0 makes a stored value fade over time.

import numpy as np

estado = np.array([8.0])
compuerta = 0.5

for paso in range(3):
    estado = estado * compuerta

print(estado[0])

Each step multiplies the state by the gate without writing anything new. After three steps, the output is exactly 1.0: the original value, 8.0, diluted down to an eighth.

The second example fixes the gate at 1.0 and uses a one-hot key to write and read an exact row, the same setup Qwenputer uses.

import numpy as np

memoria = np.zeros((8, 4))  # 8 addresses, 4 values per cell

def escribir(memoria, direccion, valor):
    memoria[direccion] = valor  # gate=1.0 and beta=1.0: direct overwrite
    return memoria

def leer(memoria, direccion):
    clave = np.zeros(8)
    clave[direccion] = 1.0  # one-hot encoding
    return clave @ memoria

memoria = escribir(memoria, 3, np.array([7, 0, 0, 0]))
print(leer(memoria, 3))

The output is [7. 0. 0. 0.]: row 3 stored the exact value and the one-hot query retrieved it without touching any other address. That’s, in miniature, the operation Qwenputer repeats thousands of times to move data between its CPU’s registers.

Real Use Cases for This Linear Memory

On that foundation, Agrawal compiled a complete RISC-V RV32IM CPU (with the Zba extension and a small unit for big-number arithmetic) into 11 of Qwen3.5’s 32 layers. The compiler translates each instruction into steps like “read this register,” “add these two numbers,” or “jump if zero,” and each step turns into attention weights and weights for the layers acting as RAM.

flowchart TD
subgraph Model["Qwen3.5 4B: 32 total layers"]
CPU["First 11 layers: compiled RISC-V CPU"]
Rest["Remaining 21 layers: Qwen3.5's original weights"]
end
CPU --> Rest

According to the original article itself, that CPU passes the official riscv-tests suite, its results match QEMU’s, and it ran 18 million instructions autonomously without a single error. Qwen3.5 doesn’t invoke this CPU as an external tool: it generates the address, value, and end-of-instruction tokens as a normal part of its text, and those tokens trigger the layers acting as hardware.

sequenceDiagram
participant Q as Qwen3.5
participant R as RISC-V CPU
participant M as Linear attention state
Q->>R: token ADDR
Q->>R: token VAL
Q->>R: token STEP
R->>M: writes VAL to row ADDR
M-->>R: confirms write
R-->>Q: instruction result

On that same foundation, the experiment runs C and MicroPython code inside the model’s forward passes, though still slowly, and the author added a rudimentary file system that uses the same one-hot address trick to simulate a disk. Two derivative projects explore the risks of this same mechanism: one shows how a hybrid model could hide part of its reasoning in the state of Qwen’s linear RNN to evade chain-of-thought supervision, since a monitor that only reads the generated text wouldn’t see anything stored directly in that memory. The other proposes the opposite: using compiled weights like this CPU as an internal auditor that the model itself can’t retrain or manipulate during inference, because that circuit wasn’t learned, it was hand-fixed.

Common Mistakes and Best Practices

• Confusing it with tool calling. Qwenputer doesn’t call an external tool or run in a sandbox: the CPU lives inside the same weights that generate the text, and the “driver” only loads the initial firmware into the state.
• Assuming Qwen3.5 already does this by default. Without the modification, the model uses a gate below 1.0, so it doesn’t store exact values, only an approximate blend of recent history.
• Expecting real-silicon performance. The author himself clarifies this isn’t a Linux-class chip: it runs slower than a normal interpreter and, for now, is limited to arithmetic and small programs.
• Treating the delta rule as if it were new. Delta-rule-based associative memory dates back to 2021; what’s new is making it exact and addressable inside a production model.

Comparison: Standard Attention, Gated Linear Attention, and Other Alternatives

The table below summarizes how each mechanism stores and retrieves information, and when it makes sense to think of them as memory.

Going Deeper: Limits and Advanced Math

The generalized delta rule Gated DeltaNet uses doesn’t just multiply by the gate: it also subtracts the projection of the current key before adding the new value, which keeps a write from completely wrecking whatever was previously stored at similar addresses. With a one-hot key that subtraction disappears, because two distinct one-hot keys are orthogonal by definition, which is why Qwenputer’s case simplifies to a clean row-by-row overwrite.

The cost of this memory also differs from standard attention: a gated linear attention head keeps a fixed-size state, while the KV-cache of full attention grows linearly with every new token. That difference is what makes it viable to use the layer as RAM: writing and reading cost the same at token 10 as at token 10,000. There’s no direct way to verify this cost from outside the model without access to the inference code, but Qwen3-Next’s architecture publicly documents which layers use linear attention and which use full attention.

The limits are real. The project itself clarifies that the CPU implements the full RV32IM instruction set but isn’t, yet, a Linux-class chip, and that running C or MicroPython inside the model’s forward passes is noticeably slower than a normal interpreter. The author also notes that the experiment, for now, stays within arithmetic and small programs: going further, like handling virtual memory or real interrupts, remains future work.

💭 Key takeaway: none of this required retraining the model. The compiler fixes the weights by hand, layer by layer, like wiring a circuit instead of teaching it to learn one.

Your next step: open the Qwenputer repository on Hugging Face or GitHub and follow the log of a simple addition instruction to watch the address, value, and end-of-instruction tokens get generated one by one.

Frequently Asked Questions

What is Qwenputer?

Qwenputer is a modified version of Qwen3.5 4B in which a RISC-V RV32IM CPU gets compiled directly into the model’s weights, using the model’s own gated linear attention as RAM memory.

Does Qwen3.5 use this exact memory by default?

No. By default, Qwen3.5’s gate is below 1.0, so the state fades with every new token. Qwenputer forces the gate to 1.0 so the memory stops being lost.

What is the delta rule?

It’s the update rule gated linear attention uses: at every token, it nudges one direction of the state toward a new value based on the write strength (beta), instead of rewriting the entire matrix.

Why doesn’t this memory’s state grow with the conversation?

Because each head keeps a fixed-size matrix, unlike the KV-cache of standard attention, which adds a row for every new token in the history.

Can real code actually run inside Qwen3.5’s weights?

Yes, experimentally: Qwenputer runs C and MicroPython programs using the CPU compiled into its layers, though more slowly than a conventional interpreter and without support for a full operating system.

References

• Divyansh Agrawal, “Qwenputer: a fully functioning, deterministic RISC-V CPU inside Qwen3.5”: the original article with the technical details of the CPU, the compiler, and the derivative experiments.
• Schlag, Irie, and Schmidhuber, “Linear Transformers Are Secretly Fast Weight Programmers,” ICML 2021: the paper that showed linear attention with the delta rule works as associative memory.
• Qwen organization on Hugging Face: repository with the public weights of the Qwen family, including Qwen3-Next.
• RISC-V, Wikipedia: general reference for the instruction set architecture Qwenputer compiles inside the model.

📱 Enjoying this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day.

Featured image: Foto de Growtika en Unsplash