⏱️ Lectura: 11 min
Nineteen times in six months, a SQLite database got corrupted inside Tailscale’s infrastructure without anyone being able to reproduce the failure in a lab. The company deployed forensic telemetry in production for months to track down the origin: a SQLite bug 16 years old, hidden in the Write-Ahead Logging (WAL) mechanism used by thousands of applications.
📑 En este artículo
The case, published on August 12, 2026 on Tailscale’s technical blog, is a rare example of a company ending up helping fix a bug at the heart of one of the world’s most widely used database engines. For any team running SQLite in production, the story leaves concrete lessons about backups, WAL mode, and corruption diagnosis.
TL;DR
- Tailscale suffered 19 corruption incidents in its SQLite database between August 2025 and early 2026.
- The root cause was a 16-year-old SQLite bug, tied to the Write-Ahead Logging (WAL) mechanism.
- Tailscale has used SQLite as the primary database for its control plane since 2022.
- Each shard runs a single Go process with exclusive access to its SQLite database (single-writer design).
- The bug couldn’t be reproduced in a lab; the team deployed forensic telemetry in production to catch it.
- During corruption events, the shard’s control plane would halt and new devices couldn’t join.
- The affected data was only configuration metadata, never private keys or network traffic.
- Tailscale confirms it has already identified, understood, and fixed the bug before publishing the report.
Introduction
Since late 2025, Tailscale’s status page showed recurring instability with no apparent pattern. The root cause lived inside SQLite, the embedded database that underpins the control plane behind every WireGuard connection between devices on a tailnet. Tracking down a SQLite bug that had gone undetected for 16 years wasn’t an afternoon’s work: it took months of forensics in production, and the company itself acknowledges that the repeated instability eroded trust among some of its users, even though most tailnets were never directly affected.
What happened
Tailscale’s control plane isn’t a monolithic service. Behind the public endpoint controlplane.tailscale.com sits a series of internal coordination shards: each tailnet lives on one shard at a time and can migrate between shards without the user noticing. Each shard runs a single Go process that exclusively opens its own SQLite database, exactly the single-writer pattern SQLite was designed for from the start.
The backup pipeline took a full copy of the SQLite file every few minutes and uploaded it to an S3 bucket, a scheme that had run without incident since early 2023. Everything changed in August 2025: a pipeline reading those backups reported an error, and running PRAGMA integrity_check against the file confirmed what nobody expected: the database was corrupted.
SQLite database corruption is possible, but uncommon in normal operation. The team repaired the database, investigated the cause, and found nothing conclusive. When the same error showed up again, and again, it became clear this wasn’t an isolated incident. In total, Tailscale logged 19 corruption incidents in six months before resolving the root-cause bug, according to its own count.
Context and history
Tailscale adopted SQLite as its primary database in 2022 because, in its own words, it’s boring technology in the good sense: well known, reliable, and used at greater scale by other companies without surprises. SQLite’s promise is simple: a single file, no server to manage, with full ACID guarantees as long as only one process writes at a time. That last point turned out to be, indirectly, the crux of the problem.
Every time a shard suffered corruption, the control plane process had to stop completely while the team repaired or restored the database. For tailnets hosted on that shard, the control plane disappeared for the entire recovery window. In the earliest incidents that window exceeded an hour; with each repetition, the team sped up the recovery process. Devices that were already connected kept talking to each other over WireGuard peer-to-peer, but couldn’t learn about network changes: a new device couldn’t join the tailnet until the shard came back online, and the web console and Tailscale’s API were inaccessible to those users.
Tailscale publishes a global incident on its status page even when it affects only a small fraction of shards, so much of its user base saw outage reports that never actually touched them. Even so, the problem’s repetition (19 times in half a year) was enough to wear down trust, beyond the real scope of each outage.
Technical details of the SQLite bug
What made this SQLite bug especially difficult was the absence of a common factor. Tailscale’s team reviewed recent changes to its low-level code and found nothing relevant: nobody had touched that layer in years. They also found no pattern across incidents: it wasn’t tied to a specific shard, a client, a particular product feature, a time of day, or a load level. Without a reliable trigger, they couldn’t synthetically reproduce the failure in a test environment.
The only way out was to instrument the real environment with passive forensic telemetry: live diagnostics, running in production, waiting to catch the corruption at the exact moment it occurred. The name the finding ended up receiving (WAL-Reset bug) points to SQLite’s Write-Ahead Logging mechanism, the mode in which writes first accumulate in a separate log file (-wal) before being applied to the main file through a checkpoint. A subtle flaw in the logic that resets that log after a checkpoint is exactly the kind of race condition that only appears under a very specific combination of timings, not under sustained high load, which explains why it stayed invisible through years of normal use.
flowchart TD
A["Tailscale Device"] --> B["controlplane.tailscale.com"]
B --> C["Coordination Shard"]
C --> D[("Shard SQLite Database")]
C --> E["Backup every few minutes"]
E --> F[("S3 Bucket")]| Mode | When to use it | Advantage | Limitation |
|---|---|---|---|
| Rollback journal (default) | Apps with few concurrent writes | Simplicity, no extra persistent files | Readers and writers block each other |
| WAL (Write-Ahead Logging) | Apps with concurrent reads and writes, like a control plane | Readers don’t block the writer and vice versa | Requires periodic checkpoints and management of the -wal file |
How to start shielding your SQLite database
You don’t need to operate at Tailscale’s scale to run into silent corruption. Here are the minimum steps to shield any SQLite database in production, whether it’s your own backend or a desktop app.
# Windows (PowerShell, via winget)
winget install SQLite.SQLite
# macOS (Homebrew)
brew install sqlite
# Linux (Debian/Ubuntu)
sudo apt install sqlite3
# Check the integrity of an existing database
sqlite3 tailnet.db "PRAGMA integrity_check;"A healthy database responds ok. Any other text lists the damaged pages or indexes and is the signal to restore from the last verified backup.
💡 Tip: Run PRAGMA integrity_check; as part of your backup pipeline, not only when something already looks wrong: it’s the only way to catch silent corruption before it spreads.
-- Enable WAL on a SQLite database
PRAGMA journal_mode=WAL;
-- Confirm it's active (should return "wal")
PRAGMA journal_mode;
-- Force a checkpoint and see how many pages moved
PRAGMA wal_checkpoint(TRUNCATE);If PRAGMA journal_mode; doesn’t return wal, the mode didn’t take effect (for example, because of a connection open in memory mode) and you need to review how the connection is opened.
db, err := sql.Open("sqlite3", "file:shard.db?_journal_mode=WAL&_busy_timeout=5000")
if err != nil {
log.Fatalf("could not open shard.db: %v", err)
}
defer db.Close()
// Single-writer: one single *sql.DB per shard, no writer pool
db.SetMaxOpenConns(1)The _busy_timeout avoids database is locked errors when a checkpoint briefly competes with a write, and limiting connections to a single one respects the single-writer pattern SQLite expects.
⚠️ Watch out: Before assuming corruption is a bug of your own, first isolate the access pattern: SQLite requires a single writer per file, and sharing the connection across processes without coordinating it is the most common cause of corruption, although in this case the origin was in SQLite itself.
Impact and analysis
The case matters beyond Tailscale because SQLite is, according to its own maintainers, probably the most deployed database in the world: it lives inside browsers, mobile operating systems, desktop applications, and increasingly, production backends that would previously have defaulted to Postgres or MySQL. A 16-year-old SQLite bug doesn’t affect only Tailscale: in theory, it affects any project using the same WAL mechanism under similar write conditions. That a single company, with enough scale and enough telemetry discipline, was able to isolate it is itself a service to the rest of the ecosystem.
It also leaves a lesson about observability: when a bug can’t be reproduced in a controlled environment, the only real alternative is to instrument production carefully and patiently, without compromising the availability of the very system being debugged. Tailscale documents that entire process, not just the final result, which is uncommon in public reports of this kind.
What’s next
Tailscale states in its post that it has already identified the bug, understood it thoroughly, and fixed it, and that it’s confident in its platform’s stability for the rest of 2026. What’s still unclear is whether SQLite’s core team will merge the fix directly into a future official release of the project, or whether Tailscale is maintaining its own patch in the meantime. Any team relying on SQLite with a similar write pattern (long-running processes, high volume of small transactions, frequent full-file backups) should check the official SQLite changelog in upcoming releases to confirm whether the fix is already included.
📖 Summary on Telegram: View summary
Try it yourself: run sqlite3 your_database.db "PRAGMA integrity_check;" against your production database today, before you need it in the middle of an incident.
Frequently Asked Questions
What exactly is the SQLite bug Tailscale found?
It’s a 16-year-old flaw tied to SQLite’s Write-Ahead Logging (WAL) mechanism, which under very specific circumstances could corrupt the database file. Tailscale publicly documented it on August 12, 2026 after investigating it for months.
Am I at risk if I use SQLite in production?
The risk is low: Tailscale needed its own scale, dozens of shards with constant writes over months, just to detect the pattern at all. Even so, running PRAGMA integrity_check periodically and keeping verified backups is a recommended practice for any serious deployment.
What is WAL mode and how does it differ from the default mode?
WAL (Write-Ahead Logging) first writes changes to a separate log file and applies them to the main file through periodic checkpoints, which lets readers and the writer avoid blocking each other. The default mode (rollback journal) is simpler but blocks reads during a write.
Did Tailscale lose user data?
According to the company, the affected databases contain only configuration metadata, such as devices and network settings, never private encryption keys or network traffic. In the earliest incidents, some recent changes didn’t persist and had to be re-entered manually.
How do I detect if my own SQLite database is corrupted?
Run PRAGMA integrity_check; against the file: a healthy database returns the word ok; any other response lists the problems found.
Where can I read the full technical report?
The original account, with the complete timeline of the 19 incidents, is published on Tailscale’s official blog.
References
- Tailscale Blog: original post about the discovery of the WAL-Reset bug in SQLite, with the complete timeline of the 19 incidents.
- SQLite.org: official documentation of Write-Ahead Logging (WAL) mode.
- SQLite.org: reference for the
PRAGMA integrity_checkcommand to detect corruption. - GitHub: Litestream: open source continuous replication tool for SQLite backups in production.
📱 Enjoying this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day. @programacion
Imagen destacada: Foto de Markus Spiske en Unsplash
0 Comments