⏱️ Lectura: 10 min
AirLLM already runs the largest open source model ever released, Kimi K3, with its 2.8 trillion parameters, on a single graphics card with less than 4 GB of memory. The July 2026 update brought that number down to 3.72 GB of VRAM, measured end to end on an RTX 6000 Ada, without resorting to quantization, distillation, or pruning of the model.
📑 En este artículo
The key lies in how AirLLM moves the weights: instead of loading the entire model onto the GPU, it pulls from disk only what’s needed for each inference step. With Kimi K3, a sparse Mixture of Experts (MoE) model, that means loading one expert at a time instead of an entire layer.
TL;DR
- AirLLM (July 2026) adds support for Kimi K3, the largest open model released so far with 2.8 trillion parameters.
- Kimi K3 runs in 3.72 GB of VRAM measured end to end on a single RTX 6000 Ada.
- AirLLM lets you run Llama 3.1 405B in 8 GB and DeepSeek-V3 (671B) in about 12 GB of VRAM.
- Version 3.0 (June 2026) added FP8 support and brought Qwen3-235B-A22B down to about 3 GB of VRAM.
- Kimi K3 requires flash-attn mandatorily, compressed-tensors, torch with CUDA 12, and transformers 4.56.x.
- Per-expert streaming (MoE) loads only the experts each token activates, not the full layer.
- AirLLM’s block-wise compression offers up to 3x more speed with almost no loss of precision.
- lyogavin’s repository has racked up 26,500 stars and 2,900 forks on GitHub.
What Happened
The AirLLM changelog recorded support for Kimi K3 in July 2026, a 2.8 trillion parameter model that the project itself describes as the largest open-weight release made public to date. The key figure is memory: 3.72 GB of VRAM, measured start to finish on a single RTX 6000 Ada.
Kimi K3 is a sparse model (sparse MoE), which changes the loading strategy. In a dense model, AirLLM pulls a full layer onto the GPU, runs it, and discards it. In an MoE like Kimi K3, each token only activates a subset of experts, so AirLLM pulls only those experts from disk instead of the entire layer. It’s the difference between loading an entire floor of a building versus only the office you’re actually going to use.
Support arrived with three requirements that aren’t optional. The model’s code demands flash-attn regardless of what’s requested when initializing it, so compressed-tensors and flash-attn need to be installed separately. It also requires a torch build with CUDA 12, since there’s no prebuilt flash-attn wheel for CUDA 13 yet. Finally, the repository pins transformers to the 4.56.x series: Kimi K3’s remote code doesn’t load on the 5.x series.
⚠️ Heads up: if you already have an environment with transformers 5.x or a torch build for CUDA 13, Kimi K3 won’t load. Set up a separate virtual environment before touching your usual setup.
Context and History
AirLLM was born in November 2023 with a concrete promise: run a 70B Llama 2 on a 4 GB GPU without quantizing it. From there, lyogavin’s project kept adding pieces one by one. In December 2023 came support for safetensors and the top ten models on the open LLM leaderboard, followed by ChatGLM, QWen, Baichuan, Mistral, and InternLM, and with version 2.0, a compression method that tripled execution speed. Version 2.5 added prefetching (pulling the next layer while the current one is still computing), with a 10% speed improvement. Version 2.6 unified everything behind a single AutoModel class, and 2.7 added AirLLMMixtral. By year’s end, 2.8.2 already let a 70B model run on macOS.
In 2024 the focus shifted to Meta’s models and to efficiency: native Llama 3 support in April, Llama 3.1 405B along with 8-bit and 4-bit quantization in July, CPU inference in August, and Qwen2.5 that same week. Then there was a big jump forward to June 2026, when version 3.0 added FP8 support and brought DeepSeek-V3 (671B) down to about 12 GB and Qwen3-235B-A22B down to about 3 GB, all behind the same AutoModel interface. Kimi K3 in July 2026 is the latest piece of that progression: from 70B in 4 GB to 2.8T in 3.72 GB in under three years.
Technical Details and Performance
The underlying mechanism is called layer streaming: instead of loading all the model’s weights into VRAM before generating, AirLLM partitions the model by layers (or by experts, in the case of an MoE), stores it on disk in sections, and pulls each section into the GPU right before it’s needed, discarding it as soon as it’s done. That’s why a 70B model, which normally requires more than 140 GB at full precision, can run on a 4 GB card: it’s never fully in memory at the same time.
On top of that streaming, AirLLM offers compression based on block-wise quantization, which, according to the project itself, can speed up inference up to 3 times with almost no loss of precision. Enabling it requires having bitsandbytes installed along with a recent version of airllm. The other documented optimization is version 2.5’s prefetching, which overlaps loading the next layer with computing the current one and delivered a 10% speed improvement over the previous version.
| Model | Parameters | Approximate VRAM with AirLLM | Available since |
|---|---|---|---|
| Llama 2/3 (base) | 70B | 4 GB | 2023/2024 |
| Llama 3.1 | 405B | 8 GB | 2024/07 |
| DeepSeek-V3 | 671B | ~12 GB | 2026/06 (v3.0) |
| Qwen3-235B-A22B | 235B (22B active) | ~3 GB | 2026/06 (v3.0) |
| Kimi K3 | 2.8T | 3.72 GB (measured on RTX 6000 Ada) | 2026/07 |
In an MoE model like Kimi K3, per-expert streaming replaces full-layer streaming: the model’s router decides which experts each token activates, and AirLLM only pulls those specific experts from disk.
flowchart TD
A["Input token"] --> B["Kimi K3 router"]
B --> C["Selects active experts"]
C --> D["AirLLM loads those experts from disk"]
D --> E["GPU runs the forward pass"]
E --> F["Result and VRAM discard"]Getting Started
Installation is the same Python library on any operating system; the only thing that changes is how you set up the virtual environment.
Linux / macOS:
python3 -m venv airllm-env
source airllm-env/bin/activate
pip install airllmWindows (PowerShell):
python -m venv airllm-env
airllm-env\Scripts\Activate.ps1
pip install airllmWith that, you can already load almost any popular Hugging Face model with the same AutoModel class, no matter which one it is:
from airllm import AutoModel
MAX_LENGTH = 128
modelo = AutoModel.from_pretrained("Qwen/Qwen3-32B")
entrada = ["What is the capital of El Salvador?"]
tokens_entrada = modelo.tokenizer(
entrada,
return_tensors="pt",
return_attention_mask=False,
truncation=True,
max_length=MAX_LENGTH,
padding=False
)
salida = modelo.generate(
tokens_entrada["input_ids"].cuda(),
max_new_tokens=20,
use_cache=True,
return_dict_in_generate=True
)
texto = modelo.tokenizer.decode(salida.sequences[0])
print(texto)For Kimi K3 you need to first install the specific dependencies and pin the versions the model requires:
# Mandatory prerequisites for Kimi K3
pip install compressed-tensors flash-attn
pip install "transformers==4.56.*"
python -c "import torch; print(torch.version.cuda)" # should print 12.x💡 Tip: if you want the extra speed from block-wise compression, install bitsandbytes before initializing the model; according to AirLLM, it can triple inference speed with almost no loss of precision.
To confirm that streaming is working and that you didn’t load the full model into VRAM, open a second terminal while you’re generating text:
watch -n 1 nvidia-smi --query-gpu=memory.used --format=csvIf memory usage stays below the gigabytes your card has instead of spiking to the model’s full size, layer (or expert) streaming is active. It’s also worth checking the transformers version before loading Kimi K3:
python -c "import transformers; print(transformers.__version__)"
Impact and Analysis
What changes with AirLLM isn’t the model itself, but who can run it. Before this technique, evaluating Kimi K3 or DeepSeek-V3 locally required a cluster of data center GPUs. With layer or expert streaming, a developer with a single mid-range card can load the same model, even if only for one-off inference rather than serving production traffic.
There’s a real cost worth stating plainly: pulling weights from disk on every generation step is slower than having the entire model resident in VRAM. AirLLM doesn’t publish a tokens-per-second figure for Kimi K3 in its documentation, so the honest way to evaluate the trade-off is to measure it yourself with the nvidia-smi command from the previous section, along with a simple timer around the generate() call. That makes AirLLM better suited to evaluation, prototyping, or batch inference than to a low-latency service with multiple concurrent users.
The flip side is that this per-expert streaming pattern, designed first for giant MoEs like Kimi K3, will probably become the de facto standard for running any large sparse model on consumer hardware, the same way layer streaming already is for dense models.
What’s Next
AirLLM’s update history (CPU in 2024, macOS in late 2023, FP8 in June 2026, per-expert streaming in July 2026) suggests the next logical step is adding support for the next big MoE model released in open weights, with the same dependency adjustments Kimi K3 required. It’s worth following the repository’s Updates section for the next leap.
📖 Summary on Telegram: View summary
Try it yourself: run pip install airllm and load a model with AutoModel.from_pretrained() right now to see how much VRAM layer streaming frees up on your own GPU.
Frequently Asked Questions
What is AirLLM?
It’s a Python library that drastically reduces the memory needed to run large language models, pulling from disk only the layers or experts needed at each step, without quantizing, distilling, or pruning the model.
Do I need to quantize the model to use AirLLM?
No. Layer streaming works with the model at its original precision. Block-wise compression is a separate option, using bitsandbytes, which adds speed but isn’t mandatory for the model to fit in little VRAM.
Why does Kimi K3 require flash-attn?
Because the model’s own code explicitly requests it, regardless of what configuration you pass to AirLLM when loading it. It’s a requirement of the model, not of the library.
Does it work on Mac or only with an NVIDIA GPU?
AirLLM added CPU inference and macOS support back in 2023 and 2024, but the VRAM figures in this article (including Kimi K3’s) correspond to execution on an NVIDIA GPU.
Is it slower than running the full model in VRAM?
Yes, generally: pulling weights from disk at every step adds latency compared to having the entire model already loaded on the card. AirLLM doesn’t publish a speed figure for Kimi K3, so it’s worth measuring on your own hardware.
What models does it support besides Kimi K3?
Among others, Qwen3, the Llama 3.x/4 family, DeepSeek V2/V3, Phi-4, and Gemma, all through the same AutoModel class.
References
- AirLLM’s official GitHub repository: source code, full changelog, and usage examples.
- airllm package on PyPI: installation page and published versions.
- Hugging Face Transformers documentation: reference for the library AirLLM uses as its base for tokenization and generation.
📱 Enjoying this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day.
0 Comments