Sep 20, 2026 12:00:00

How I found a SQLite bug that had been left unaddressed for 16 years.

Tailscale , a VPN construction and network connectivity service provider, investigated the reason for its service instability for several months and discovered that the cause was a bug that had been latent in its SQLite database for 16 years. Tailscale has published an explanation on its blog about 'what the problem was,' 'how they addressed it,' and 'how they contributed to discovering the SQLite bug.'

How Tailscale helped find the SQLite WAL-Reset bug
https://tailscale.com/blog/sqlite-wal-reset-bug

Tailscale has been using SQLite as its primary database since 2022. When a user accesses a Tailscale endpoint, they connect to the 'control plane,' which is composed of multiple 'shards.' The secure private network, the 'tailnet,' rests on one of these shards, but there is a mechanism to seamlessly migrate between shards depending on the situation. Each shard has an SQLite database that stores all the information about the tailnet it manages. A single Go process has exclusive access to the SQLite database and handles the tailnet's control plane.

The backup mechanism takes a full snapshot of the database every few minutes and uploads the entire SQLite file to an

Amazon S3 bucket. The configuration had been running without issue since early 2023, but problems began in August 2025 when database corruption was reported in the S3 backup. There were 19 database corruptions in six months, but the impact was limited to control plane configuration data. However, each time a corruption occurred, the control plane process had to be stopped to repair or restore the database, resulting in downtime when the control plane containing the affected shard was unavailable.

To resolve the issue, we first checked recent changes, but found no changes that seemed related to the bug. In particular, all the low-level code that interacts with SQLite was written several years ago, and since no bugs had occurred, the code had not been modified. We also thoroughly examined the code as a precaution, but found no code that could cause data corruption. Furthermore, the trigger for the problem was unknown, so we were unable to reproduce the bug and had to rely on forensic telemetry in the live environment. To make matters worse, the incident sometimes occurred every few hours, and other times nothing happened for weeks. Judging that this was not a problem that could be easily fixed, Tailscale entered into a professional support agreement with the SQLite developers and began working together to verify the problem.

The platform remained operational while we investigated the root cause, and we took the following steps to automate recovery and minimize downtime:

- Configure the shards to immediately hard stop if the database becomes corrupted.
Implement automatic backups.
- Improve operational procedures and on-call training.

These efforts allowed us to reduce response times to less than one hour and also provided clues to the discovery of unexpected problems.

Seeking a way to recover services without data loss or risk, Tailscale decided to build a transaction log pipeline, streaming all SQL statements that modify the database to a log file. The aim was that reproducing these transactions against the most recent healthy backup would restore the database to its latest state, safely avoiding database corruption.

The implementation of the transaction log pipeline not only worked as expected, but also provided clues to bugs. In two incidents, we failed to reproduce the transaction logs, and upon closer investigation, we discovered that data that should have been written and committed by one transaction was not appearing in subsequent transactions.

The situation that has become clear suggests that there may be a problem somewhere in the SQLite checkpointing process. SQLite databases are made up of a series of 'pages,' or small blocks of information. When updating the database, some of the pages need to be replaced with ones that contain the updated information. To improve performance and concurrency, SQLite runs with 'Write-Ahead Logging (WAL)' enabled. This means that new pages are not written directly to the database file, but are first written to the WAL.

New pages cannot be written to the WAL indefinitely; at some point, they need to be copied back to the main database file. This process is called a 'checkpoint.'

Normally, end users and developers do not need to explicitly execute the checkpointing process, as SQLite automatically performs checkpoints. However, the control plane takes a non-standard approach of manually controlling the checkpointing process to ensure fast and consistent backups.

One clue was that metrics showed SQLite was copying more pages from the WAL than were actually available when the database was corrupted. SQLite consists of several layers. The top layer is the parser and code generator, which translates SQL statements into SQLite's internal data structures. These data structures are passed to a pager, which divides them into individual pages that are written to disk. The actual writing to disk is handled by the OS interface, or 'virtual file system'. This approach allows for replacing different layers with different implementations, or wrapping existing layers to extract more information.

To help diagnose problems, SQLite developers have created a virtual file system wrapper that writes additional trace information and logs changes to the database. This wrapper includes the 'tmstmpvfs shim,' a debug and diagnostic

VFS shim that extends and wraps the SQLite virtual file system (VFS) layer.

The SQLite development team was finally able to discover and fix a bug thanks to additional logs left by the tmstmpvfs shim following a database corruption that occurred immediately after deploying the wrapper to the production environment. The bug was caused by a rare data race that occurred between checkpoints and write transactions, where the checkpoint process would become confused and data would be lost if a write transaction occurred during the checkpoint process.

SQLite developers named the discovered bug the 'WAL-Reset bug' and estimated that it had been latent in SQLite for at least 16 years. The reason Tailscale frequently encountered the bug, despite it being a rare occurrence, was that Tailscale manually controlled the checkpoint creation process and created checkpoints very aggressively.

SQLite developers released SQLite 3.52.0, which fixed the WAL-Reset bug, but another problem arose, so they withdrew 3.52.0 and re-released 3.51.3, which only included the fix for the WAL-Reset bug. Finally, SQLite 3.53.0 was released, resolving all issues. After applying the fix, the final step was to verify that the database would no longer become corrupted even if the 'write transaction and WAL reset conflict' that caused the bug occurred. After about two months, the long-awaited alert finally occurred, confirming that database corruption no longer occurred.

Ultimately, Tailscale has been operating without any database incidents for four months after confirming that the fix for the WAL-Reset bug was effective. The experience with the WAL-Reset bug highlighted the risks of performing 'common techniques' in a non-standard way and reinforced the reliability of standard configurations and paths. The resolution of the incident was a major collaborative effort for both Tailscale and SQLite developers, and it has led to long-standing bug fixes in SQLite and improvements to Tailscale's backup and recovery processes.