Gigatoken Tokenizes at 24.53 GB/s: Up to 989 Times Faster Than HuggingFace

⏱️ Lectura: 10 min

The gigatoken tokenizer, published on GitHub by developer marcelroed, processes the entire OpenWebText corpus (11.9 GB) in under half a second: 24.53 GB/s with GPT-2, compared to HuggingFace Tokenizers’ 24.8 MB/s on the same server. This nearly thousandfold difference occurs even though HuggingFace Tokenizers and tiktoken already run on pure multithreaded Rust.

📑 En este artículo

1. TL;DR
2. Introduction to gigatoken
3. What happened with the gigatoken tokenizer
4. Context and history
5. Technical details and performance of the gigatoken tokenizer
6. Getting started
7. Impact and analysis
8. What’s next
9. Frequently Asked Questions
   1. What is gigatoken?
   2. How fast is it compared to HuggingFace Tokenizers?
   3. Does it work with any HuggingFace tokenizer?
   4. How do you install gigatoken?
   5. Is compatibility mode as fast as the native API?
   6. Does gigatoken replace OpenAI’s tiktoken?
10. References

TL;DR

• Gigatoken reaches 24.53 GB/s tokenizing GPT-2 on a 144-core AMD EPYC 9565, 989 times faster than HuggingFace Tokenizers.
• Against tiktoken, OpenAI’s library, the difference is 681 times (36.0 MB/s for tiktoken versus 24.53 GB/s for gigatoken).
• On a 16-core Apple M4 Max it reaches 8.79 GB/s with GPT-2, 1,268 times faster than HuggingFace on the same machine.
• It supports more than 15 tokenizer families: Llama 3, Qwen 3, DeepSeek V3, GLM 5, Kimi K2, GPT-OSS, and Nemotron 3, among others.
• It installs the same way on Linux, macOS, and Windows with pip install gigatoken, with no need to compile Rust manually.
• It offers a direct compatibility mode (as_hf() and as_tiktoken()) to replace existing tokenizers without changing the rest of the code.
• The speedup isn’t uniform: families like Gemma 4 and Mistral 7B v0.3 only perform 7 to 14 times faster, not close to 1,000.

Introduction to gigatoken

Gigatoken is an open source Rust project, with Python bindings, presented as a drop-in replacement for HuggingFace Tokenizers and tiktoken. The repository, maintained by user marcelroed, has accumulated 1,200 stars, 46 forks, and 357 commits on the main branch. Its core promise is simple: tokenize text at gigabytes-per-second speeds instead of megabytes per second, without changing the final tokenization output.

Besides the code in src/, the project includes files like design_doc.md and pretokenizer_optimization_log.md at the root of the repository, suggesting a step-by-step documented development process of how each stage of the pretokenizer was optimized.

What happened with the gigatoken tokenizer

The project’s README publishes two complete benchmark tables: one run on a dual-socket AMD EPYC 9565 server (144 cores) and another on a laptop with an Apple M4 Max (16 cores). Both use the same corpus, OpenWebText’s owt_train.txt, with 11.9 GB of plain text, and measure how many bytes per second each tokenizer processes when encoding that full file with different vocabularies: GPT-2, Llama 3, Qwen 3, DeepSeek V3, GLM 5, Kimi K2, GPT-OSS, among fifteen others.

On the EPYC server, gigatoken tokenizes GPT-2 at 24.53 GB/s. HuggingFace Tokenizers, evaluated on the same machine and the same file, reaches 24.8 MB/s: gigatoken is 989 times faster. Against tiktoken, OpenAI’s official library, the difference is 681 times (36.0 MB/s for tiktoken versus 24.53 GB/s for gigatoken).

Context and history

Tokenization is the first step in any language model training pipeline: it converts raw text into the sequence of numerical identifiers the neural network consumes. When the training dataset weighs hundreds of gigabytes or terabytes, that first step can become a bottleneck that delays hours of GPU computation while waiting for data.

HuggingFace Tokenizers, the reference library of the Transformers ecosystem, is already written in Rust and already multithreaded: it’s not a naive pure Python implementation. The same goes for tiktoken, OpenAI’s library for the GPT family’s vocabularies. Gigatoken’s own README clarifies this upfront: “Note that both HF tokenizers and tiktoken are already running multithreaded Rust!”. The performance gap, then, doesn’t come from swapping Python for Rust (that was already done), but from another layer of optimization underneath.

💭 Key point: HuggingFace Tokenizers and tiktoken are already multithreaded Rust. Gigatoken’s advantage doesn’t come from rewriting in a faster language, but from reducing the overhead around the tokenization core.

Technical details and performance of the gigatoken tokenizer

Gigatoken offers two ways to use it. Compatibility mode wraps an existing HuggingFace or tiktoken tokenizer with as_hf() or as_tiktoken(), so current code keeps working without changes, reproducing the original tokenizer’s exact output. The project itself warns that this mode carries a non-negligible performance cost compared to its own native API, although it remains much faster than the original library in any scenario.

Gigatoken’s native API, with gt.Tokenizer and a TextFileSource, lets the Rust code read files directly from disk, without passing text strings across the Python/Rust boundary, which is where time is lost in compatibility mode. That’s the structural difference behind the numbers in the table.

Tokenizer	Gigatoken	HF Tokenizers	tiktoken	vs HF
GPT-2	24.53 GB/s	24.8 MB/s	36.0 MB/s	989×
Phi-4	24.00 GB/s	29.9 MB/s	,	801×
GPT-OSS	23.96 GB/s	49.7 MB/s	42.8 MB/s	482×
Llama 3 / 3.1 / 3.2	22.15 GB/s	48.5 MB/s	,	457×
DeepSeek V3 / R1 / V4	19.69 GB/s	26.2 MB/s	,	750×
GLM 5	20.97 GB/s	74.8 MB/s	,	280×
Gemma 4	4.82 GB/s	334.1 MB/s	,	14×
Mistral 7B v0.3	3.57 GB/s	354.7 MB/s	,	10×

The full EPYC table includes twenty tokenizer families; the pattern repeats across all of them: tokenizers with classic BPE-style vocabularies, like GPT-2, Llama, DeepSeek, or Qwen, perform between 400 and 990 times faster, while families with heavier pretokenization rules, like Gemma or Mistral, stay in a range of 7 to 14 times. It’s a real difference, but far from the “1000x” headline.

On consumer hardware, a 16-core Apple M4 Max, gigatoken tokenizes GPT-2 at 8.79 GB/s: 1,268 times faster than HuggingFace Tokenizers, which reaches 6.9 MB/s, and 140 times faster than tiktoken, which reaches 62.8 MB/s. With Qwen 2 / 2.5, it hits 6.37 GB/s versus HuggingFace’s 5.8 MB/s, a difference of 1,105 times, the largest in the entire M4 Max table.

flowchart TD
A["Text file (owt_train.txt)"] --> B["TextFileSource in Rust"]
B --> C["Gigatoken tokenization core"]
C --> D[("Output tokens")]
subgraph "Native API"
B
C
end

Getting started

Gigatoken installs the same way on all three platforms because it’s distributed as a Python package with the core already compiled in Rust:

# Linux / macOS (bash or zsh)
pip install gigatoken

# Windows (PowerShell)
pip install gigatoken

# Windows (cmd.exe)
pip install gigatoken

With the package installed, compatibility mode lets you replace an existing tokenizer without touching the rest of the pipeline:

import gigatoken as gt
from transformers import AutoTokenizer

hf_tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
tokenizer = gt.Tokenizer(hf_tokenizer).as_hf()

tokens = tokenizer.encode_batch([
    "The model processes the training corpus",
    "and generates the identifiers for each token"
])

To squeeze out all the performance, the native API avoids the intermediate step through Python objects:

import gigatoken as gt

tokenizer = gt.Tokenizer("Qwen/Qwen3-8B")
file_source = gt.TextFileSource(
    ["corpus_entrenamiento.txt"],
    separator=b"<|endoftext|>"
)
tokens = tokenizer.encode_files(file_source)

This is the path that avoids the overhead of passing data across the Python/Rust boundary: the file is read directly from the Rust side, with maximum parallelism across available cores.

💡 Tip: before migrating an entire pipeline, run the compatibility mode on a small batch and compare it token by token against your current tokenizer’s output: the project documents specific work to match that output byte for byte with HuggingFace Tokenizers.

Impact and analysis

For a team training or fine-tuning its own model on a corpus of hundreds of gigabytes, tokenization time stops being a trivial step when the library takes minutes or hours to prepare the data before the GPU starts working. Going from megabytes per second to gigabytes per second turns that step from hours into seconds on large datasets, as long as the rest of the pipeline (disk reads, shard mixing, writing the resulting tokens) doesn’t become the new bottleneck.

The most honest limitation is in the benchmark table itself: the gain isn’t uniform. Families like Gemma 4, Mistral 7B v0.3, or Gemma 1, with more complex pretokenization rules, perform between 7 and 14 times faster, not close to a thousand. Anyone evaluating gigatoken for their own vocabulary should run the benchmark with their specific tokenizer before assuming the largest number in the table applies.

The other cost to consider is compatibility mode: exactly reproducing the output of HuggingFace Tokenizers or tiktoken requires additional logic that reduces performance compared to the native API. For teams that already have mature pipelines built on those libraries, compatibility mode is the reasonable entry point, even though it doesn’t deliver the table’s highest number.

What’s next

The repository doesn’t publish a public roadmap in the README beyond the code itself: files like design_doc.md and pretokenizer_optimization_log.md, visible in the repository tree, suggest the project documents its optimization decisions internally as it adds support for new tokenizer families. The current list already covers recent generations like GLM 5, Qwen 3.5/3.6, and Nemotron 3, indicating active maintenance beyond the initial release.

What still isn’t publicly documented, according to the README itself, is behavior under workloads other than tokenizing a complete plain text file: incremental streaming, tokenization at inference time (where per-request latency matters more than aggregate throughput), or support for dataset formats like Parquet or WebDataset.

📖 Summary on Telegram: View summary

Try it yourself: run pip install gigatoken and compare it against your current tokenizer on your own corpus before migrating anything to production.

Frequently Asked Questions

What is gigatoken?

It’s a tokenization library for language models written in Rust, with Python bindings, published by developer marcelroed on GitHub. Its goal is to replace HuggingFace Tokenizers or tiktoken without changing the rest of the code.

How fast is it compared to HuggingFace Tokenizers?

In the published benchmark with GPT-2 on an AMD EPYC 9565 server, gigatoken reaches 24.53 GB/s versus HuggingFace Tokenizers’ 24.8 MB/s, a difference of 989 times. On an Apple M4 Max it reaches 8.79 GB/s, 1,268 times faster than HuggingFace on the same machine.

Does it work with any HuggingFace tokenizer?

Compatibility mode (as_hf()) wraps an existing HuggingFace tokenizer and aims to reproduce its exact output. The README documents explicit support for more than 15 families, including Llama 3, Qwen 3, DeepSeek V3, GLM 5, GPT-OSS, and Kimi K2.

How do you install gigatoken?

With pip install gigatoken, the same way on Linux, macOS, and Windows, because the Rust core already comes compiled inside the Python package.

Is compatibility mode as fast as the native API?

No. The project itself warns that compatibility mode with HuggingFace or tiktoken carries a non-negligible performance cost compared to gigatoken’s native API, which avoids passing data across the Python/Rust boundary.

Does gigatoken replace OpenAI’s tiktoken?

It offers an as_tiktoken() mode to use it as a drop-in replacement. In the GPT-2 benchmark on EPYC, gigatoken is 681 times faster than tiktoken, which processes 36.0 MB/s.

References

• gigatoken repository on GitHub: source code, complete benchmarks, and usage documentation.
• HuggingFace Tokenizers: the reference library gigatoken is compared against.
• tiktoken: OpenAI’s tokenizer used as a second comparison point in the benchmarks.
• Transformers documentation: context on how tokenizers integrate into training pipelines.

📱 Enjoy this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day.

Imagen destacada: Foto de Tyler en Unsplash