Tinyworldformer: a game AI fits in 520 KB of RAM

⏱️ Reading time: 13 min

A $6 microcontroller just ran, in real time, a compressed world model that invents video games frame by frame. The tinyworldformer project runs inside 520 KB of RAM on a Raspberry Pi Pico 2W, with no GPU, no cloud, and no internet connection.

📑 En este artículo

1. TL;DR
2. What is a compressed world model?
3. Why it matters
4. How tinyworldformer works under the hood
   1. Patch tokenization
   2. The looped transformer
   3. Quantization and core allocation
5. Practical examples and how to get started
   1. How to try it on your machine
6. Real use cases
7. Common mistakes and best practices
8. Comparison with alternatives for generating content with AI
9. Going deeper: why the loop saves memory and where the limit shows up
10. Frequently Asked Questions
    1. Which microcontroller runs tinyworldformer’s compressed world model?
    2. How many parameters does the looped transformer have for each game?
    3. Does training tinyworldformer before quantizing it require a GPU?
    4. What sets the actor network apart from the embedded AI that generates the frames?
    5. Can tinyworldformer run on a microcontroller other than the Pico 2W?
11. References

Its author pulled this off by splitting the work into two neural networks per frame. One generates the next game frame, the other decides which button to press, and both run at the same time on the same chip.

TL;DR

• Int8 quantization and shared weights shrink the frame generator to between 26,000 and 131,000 parameters depending on the game.
• A transformer block applied 1 to 4 times replaces a stack of separate layers and saves memory.
• The actor network, separate from the generator, weighs only 9,000 to 16,000 parameters in float32.
• The RP2350 splits the work: core 1 runs both networks, core 0 handles USB and the display.
• The SheepSeb/tinyworldformer repository lets you play all five real games from the terminal with uv.

What is a compressed world model?

A compressed world model is a generative neural network reduced in size and precision to run without a GPU: tinyworldformer implements it as an 8-bit quantized looped transformer that predicts the next frame of a video game and fits inside the 520 KB of RAM on a $6 microcontroller.

The author’s starting idea was simple. Looped transformers reuse the same block of weights multiple times instead of stacking separate layers, so they take up much less memory by design. If that saving was enough, a full generative model could fit on a pocket-sized chip.

Why it matters

Most video or game generative models depend on a GPU with several gigabytes of memory. Tinyworldformer flips that equation: it runs on a chip meant for thermostats and toy drones. That matters for the field known as edge computing, where the goal is to move inference to the device instead of relying on a remote server.

A model that runs at the edge needs no internet connection, pays no network latency, and exposes no data to an external server. Here the benefit also serves as a proof of concept. If a generative network fits in 520 KB, the question stops being whether edge AI can generate content and becomes how much content it can generate within that memory budget.

The cost angle matters as much as the technical one. A Raspberry Pi Pico 2W costs a fraction of what any GPU capable of running a full transformer costs, and it consumes orders of magnitude less energy because it has no video memory or active cooling. For hobby projects, education, or TinyML research, that cost difference decides whether an experiment happens at all.

How tinyworldformer works under the hood

Tinyworldformer splits the problem into two networks with distinct jobs. The first, the frame generator, receives the current frame and the player’s action and predicts the next frame along with the reward. The second, the actor network, only looks at the frame and decides which action to take. Neither ever sees the real game once a session starts: the generator only receives actions from the actor, and the actor only sees frames invented by the generator.

sequenceDiagram
    participant Actor as Actor network
    participant Generator as Frame generator
    Actor->>Generator: sends action
    Generator-->>Actor: delivers generated frame and reward
    Note over Actor,Generator: the generator never sees the real game

Patch tokenization

Before entering the transformer, each frame is cut into square patches: 2×2 cells in Breakout, 4×4 in Doom-lite, Flappy Bird, and Pong, 6×6 in Pac-Man lite. That produces between 36 and 80 tokens per game according to the repository. Each token adds a learned vector per cell, with a different table based on the position inside the patch to preserve order, plus a position vector and an action-taken vector.

A simplified Python example shows the same idea with a tiny 4×4 frame split into 2×2 patches:

import numpy as np

frame = np.array([
    [1, 1, 2, 2],
    [1, 1, 2, 2],
    [3, 3, 4, 4],
    [3, 3, 4, 4],
])

def to_patches(frame, patch=2):
    h, w = frame.shape
    tokens = []
    for i in range(0, h, patch):
        for j in range(0, w, patch):
            tokens.append(frame[i:i+patch, j:j+patch].flatten())
    return np.array(tokens)

print(to_patches(frame))

The literal output is a matrix of 4 tokens, one for each homogeneous region of the frame:

[[1 1 1 1]
 [2 2 2 2]
 [3 3 3 3]
 [4 4 4 4]]

The looped transformer

Instead of stacking, say, eight separate transformer blocks, tinyworldformer uses a single block (attention plus a feedforward network with ReLU) and applies it between 1 and 4 times in a row, adding the original input before each pass. The output of each pass is trained against the actual next frame, so the number of passes at runtime becomes a direct trade-off between speed and accuracy. On the Pico 2W, to keep a reasonable frame rate, the project uses two passes.

flowchart TD
    A["Frame t + action"] --> B["Tokenize into patches"]
    B --> C["Transformer block (shared weights)"]
    C --> D{"Repeat 1 to 4 times"}
    D -->|"another pass"| C
    D -->|"final pass"| E["Frame t+1 and reward"]

Quantization and core allocation

The frame generator is quantized to int8, while the actor network stays in float32 because it’s already small, between 9,000 and 16,000 parameters, and precision matters more there for choosing the right action. The RP2350 on the Pico 2W has two cores: core 1 runs the generator and the actor, core 0 handles USB and the display, according to the repository’s documentation.

This separation between generator and actor allows each network to be optimized independently. The generator, the heavy part, gets all the quantization, while the actor, already lightweight, stays untouched in float32.

flowchart TD
    A["RP2350 dual-core"] --> B["Core 0: USB and display"]
    A --> C["Core 1: inference"]
    subgraph Inferencia["Inference core"]
    C --> D["int8 frame generator"]
    C --> E["float32 actor network"]
    end

Practical examples and how to get started

The repository includes five real games written in a few lines of numpy, which lets its author compare each generated frame against the game’s actual physics. The following table summarizes memory and compute cost for each one, with data published in the tinyworldformer repository:

The exact frames column measures the percentage of transitions where every cell in the predicted frame matches the real frame. Doom-lite has the lowest number, 83.6%, but not because it plays poorly: its first-person view has 768 cells, and a single mispainted shadow pixel already marks the whole frame as incorrect under this metric.

How to try it on your machine

The project uses uv as its Python package manager, so there’s no need to install dependencies by hand one by one. On Linux or macOS:

curl -LsSf https://astral.sh/uv/install.sh | sh
git clone https://github.com/SheepSeb/tinyworldformer
cd tinyworldformer
uv run python -m tinyworldformer.games.play --game pacman

On Windows, the only step that changes is installing uv, from PowerShell: powershell -c "irm https://astral.sh/uv/install.ps1 | iex", then the same git clone and uv run commands.

The play command also accepts breakout, doomlite, flappy, or pong instead of pacman. There’s no text output to show: uv downloads the dependencies declared in pyproject.toml and opens a graphical window with the actual game running on numpy, the same one the frame generator learned to imitate.

💡 Tip: running the real game first with play before looking at the trained weights helps you understand what the quantized generator is trying to imitate, cell by cell.

Real use cases

Tinyworldformer doesn’t aim to replace a game engine. Its value lies in what it demonstrates: a generative model trained on recorded gameplay can, once quantized and compressed, run on the same chip that today handles temperature sensors or cheap toy drones. The immediate use case is educational and TinyML research, but the recipe (tokenize into patches, share weights across passes, quantize only the heaviest part) can be transferred to other small generators, such as a sensor simulator for testing an embedded controller without real hardware connected.

The repository also adds a streaming reinforcement learning agent that learns to play inside the dream generated by the model itself, instead of just replaying an already trained policy. Any button on the device hands control back to the human user, so the same board works both for watching the AI play on its own and for playing yourself inside the world it invents.

Common mistakes and best practices

• Confusing pixel accuracy with playability. An 83.6% exact-frame rate sounds bad until you understand the metric: a single different shadow pixel already counts as a total frame failure, even though the game state (enemy position, life, score) remains correct.
• Quantizing everything equally. Applying int8 to the actor network, which already has fewer than 16,000 parameters, would bring little memory savings and risk the policy’s accuracy. Quantization pays off where the model is large, not where it’s already small.
• Picking the wrong number of loop passes. A single transformer pass is faster but leaves visible artifacts, an enemy that shouldn’t be there, for example. Four passes fix almost all the error but cost more compute time per frame. Two passes was the point the project chose for the Pico 2W.
• Verifying against the real physics, not just watching the generated video. Having the real version of the game in numpy, as this repository does, is what lets you measure the error frame by frame instead of trusting that it looks right.

Comparison with alternatives for generating content with AI

The middle row describes tinyworldformer. It doesn’t try to compete on visual fidelity with a video generator trained on a GPU; it aims to keep the game playable within a tiny memory budget.

Going deeper: why the loop saves memory and where the limit shows up

The underlying reason a looped transformer weighs less is simple arithmetic: a traditional N-layer transformer stores N separate sets of weights, while the loop stores a single set of weights and reuses it N times. The cost paid in exchange is compute time (each extra pass costs the same multiply-accumulates as the previous one) rather than memory, and on a microcontroller memory is usually the scarcer of the two resources.

Int8 quantization adds a second, independent saving: each weight goes from taking up 4 bytes in float32 to taking up 1 byte, at the cost of rounding each value to one of 256 possible levels. The following Python example shows that rounding with four arbitrary weights:

import numpy as np

weights = np.array([0.42, -0.87, 1.35, -1.90], dtype=np.float32)
scale = np.max(np.abs(weights)) / 127
quantized = np.round(weights / scale).astype(np.int8)
reconstructed = quantized.astype(np.float32) * scale

print(quantized)
print(np.round(reconstructed, 4))

The literal output is:

[  28  -58   90 -127]
[ 0.4189 -0.8678  1.3465 -1.9   ]

The original weight of 1.90 reconstructs exactly because it defined the scale; the other three carry a small rounding error. That error, multiplied across thousands of weights and several loop passes, helps explain why Pac-Man lite, the most complex game with 131,000 parameters, has a lower exact-frame percentage than Flappy Bird. There’s no direct way to verify from outside, without access to the checkpoint, how much of that error comes from quantization and how much from the game’s size; the closest thing the repository itself offers is comparing the exact-frame percentage per game in the table above.

The RP2350’s FPU matters here. Without a hardware floating-point unit, dequantization (converting from int8 back to real values before each attention operation) would be much slower, because the chip would have to emulate those operations in software.

💭 Key takeaway: the memory savings from the loop and the savings from quantization are independent of each other; tinyworldformer combines them, but each could be used separately in another project.

Your next step: clone the repository, run uv run python -m tinyworldformer.games.play --game breakout, and read that game’s file inside src/tinyworldformer/games/ to see how many lines of numpy are needed for a fully playable physics.

Frequently Asked Questions

Which microcontroller runs tinyworldformer’s compressed world model?

A Raspberry Pi Pico 2W, built on the RP2350 chip, which has two cores and a floating-point unit. Core 1 runs the frame generator and the actor network; core 0 handles USB and the display.

How many parameters does the looped transformer have for each game?

Between 26,000 and 131,000, depending on the game’s complexity: Breakout is the lightest and Pac-Man lite the heaviest, according to the table published in the repository.

Does training tinyworldformer before quantizing it require a GPU?

The repository doesn’t detail the exact training hardware, but the frame generator is trained on recorded gameplay from the real game, a process normally done outside the microcontroller and before quantizing the weights for deployment on the Pico 2W.

What sets the actor network apart from the embedded AI that generates the frames?

The actor network is a much smaller MLP, between 9,000 and 16,000 parameters, that stays in float32 because its task (deciding an action from a frame) tolerates quantization error worse than generating the frame itself.

Can tinyworldformer run on a microcontroller other than the Pico 2W?

The repository is built specifically for the RP2350 on the Pico 2W, including a Rust folder for the device firmware. Porting it to another chip would require at least an FPU and enough memory for the attention buffers, but the project itself doesn’t document any other tested targets.

References

• GitHub: SheepSeb/tinyworldformer: original repository with the code, the numpy games, and the size and accuracy data for each model.
• Raspberry Pi: official manufacturer of the Raspberry Pi Pico 2W and the RP2350 chip used in the project.
• Wikipedia: Edge computing: context on moving AI inference to the device instead of a remote server.
• Wikipedia: Quantization: mathematical foundation of the quantization used to convert weights from float32 to int8.

📱 Enjoying this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day. @programacion

Featured image: Foto de Random Thinking en Unsplash