Architecture
Architecture
How the 28 crates fit together, how concurrent writers actually work, and which of the more ambitious ideas are running today versus still on the bench. Every section is labeled.
The Layers
Each layer is one or more crates, and Cargo enforces the direction of dependencies. The parser can't reach into the pager, and the B-tree can't call the planner.
Foundation
Storage
SQL
Extensions
Integration
fsqlite-core ties everything into a Connection, and fsqlite is the crate you depend on. Around them: the fsqlite shell, metrics and tracing, an optional C ABI shim, and an experimental WebAssembly build.Verification
The Live Engine
What runs when you call execute today: the query path through the crates, and the part of FrankenSQLite that is both new and live, many writers in one process with serializable isolation.
From SQL to a Row
LiveOne point lookup, followed through the workspace. The parser builds an AST, code generation in fsqlite-core and fsqlite-vdbe picks the access path and emits bytecode, the VM walks the B-tree through the pager and MVCC visibility rules, and the WAL layer decides whether the newest copy of a page is in the log or the main file. Step through it below.
MVCC, Page by Page
LiveSnapshots
Every transaction reads from a , identified by the highest commit sequence number it may see. Writers produce new page versions instead of overwriting ones a reader might be using, so readers never block and never see a half-finished write.
Concurrent writers
Plain BEGIN is promoted to BEGIN CONCURRENT. There is no fixed writer cap; open one connection per thread. A writer locks a page the first time it writes it, and if another transaction holds it, the call fails fast instead of waiting. No waiting means no wait-for cycles, so page locks can't deadlock.
Commit
Statement execution and private page changes overlap freely. Commit takes a short registry guard, runs and validation, writes through the pager, and publishes the result to the commit index. The loser of a same-page race gets SQLITE_BUSY_SNAPSHOT.
Cleaning up versions
Versions older than the oldest snapshot anyone still holds can go. The engine keeps the newest version of each page at or below that horizon, retires the rest, and frees them in batches with epoch-based reclamation once no reader can be looking at them.
Time-travel queries
PartialSELECT ... FOR SYSTEM_TIME AS OF COMMITSEQ n (or a timestamp) works on :memory: databases, which keep a ring of up to 256 snapshots taken at each commit. File-backed databases return an explicit error rather than current data. A .fsqlite-history sidecar for file-backed history is designed but not built.
The Commit Path
LiveStatement execution and page changes run on each connection's own thread, in parallel. Commit is where they meet: the committing connection takes a short per-database guard, runs first-committer-wins and SSI validation, writes its frames through the pager into the WAL, and publishes the result to the commit index, all before releasing the guard. Holding it across all three steps closes a race where a peer could claim a freshly allocated page before the allocation became visible.
The design also calls for a dedicated that batches commits and sequences fsyncs. On the default path that service is still scaffolding. A feature-gated batching service for native-mode commits has just landed on main, but nothing in the public connection drives it yet. The visualization below models the live path, including validation failures that send a connection back to retry.
The WAL Index
LiveWith a write-ahead log, the newest copy of a page might be in the WAL rather than the main file. SQLite keeps a hash table in the -shm shared-memory file that maps page numbers to WAL frames, so a reader can find the right frame without scanning the log.
FrankenSQLite reads and writes that structure in SQLite's own format and honors its reader marks and checkpoint rules. Even so, don't point FrankenSQLite and stock SQLite at the same file at the same time; hand a database from one to the other after a checkpoint. Type a page number below to watch the lookup probe the table.
The Page Cache
LiveA plain LRU cache does badly on databases: one big table scan pushes out the pages everything else keeps using. FrankenSQLite's pager uses S3-FIFO by default, which admits new pages to a small probationary queue so one-off scan pages leave quickly.
An policy is implemented in the pager crate too, but connections always open with S3-FIFO and there is no PRAGMA or option to switch. ARC balances a recency list against a frequency list and keeps “ghost” entries for recently evicted pages to learn which side deserves more room. The demo below shows ARC's four lists at work.
Byte-for-Byte File Format
LiveCompatibility comes down to details like this one. SQLite stores rowids, record header sizes and serial types as variable-length integers: one byte for small values, up to nine for the largest. FrankenSQLite encodes and decodes them exactly the same way, along with the rest of the record and B-tree page format, so files move between it and stock sqlite3 without conversion.
Drag the slider below to see how integers of different sizes map to byte widths.
Built, Partial, or Designed
These are the ideas the project is known for. Some have working code that isn't connected to the default runtime yet; some are partly wired; some are designs with pieces landing. Each section says which.
The Safe Merge Ladder
Built, not wiredPage-level MVCC has one built-in weakness. Two transactions that insert different rows into the same leaf page conflict, even though their changes don't overlap in any meaningful way. Today the second committer simply retries. The merge ladder is the planned way to let both commit when that is provably safe.
The code is in fsqlite-mvcc and tested, but the live commit path doesn't call it and the intent log isn't recorded during writes yet. Tracked as bd-3d5y3 and bd-p4dcv.
Rebase the intent
Each writer logs what it meant to do at the B-tree level (“insert rowid 42”). On conflict, that log against the winner's page. If the B-tree invariants and constraints still hold, commit.
Structured patch
If the two transactions touched different cells (compared by cell key, not by byte range), merge the cell changes, serialize any page-header changes, and re-check the page.
Abort and retry
A real conflict. The loser gets SQLITE_BUSY_SNAPSHOT. This is the only rung the live engine uses today.
What's deliberately missing: merging raw byte ranges. Two writes can touch disjoint bytes and still collide, for example when one transaction moves a cell and the other edits the cell's old location. A byte-level merge would silently drop the second update, so the design forbids it for B-tree, overflow, freelist and pointer-map pages.
RaptorQ Repair Symbols
Partial(RFC 6330) is a fountain code. It splits data into K source symbols and can generate as many as you like; almost any K symbols, plus a small margin, are enough to rebuild the original. The math happens in : a fast peeling decoder handles most symbols and finishes the rest.
In the engine today, file-backed connections encode durable WAL ranges into a -wal-fec sidecar in the background after each WAL sync (PRAGMA raptorq_repair_symbols, default 2, sets the budget). On Unix, an explicit repair_and_open call can rebuild damaged frames from it and write a verified backup. Ordinary opens don't do that yet, so default recovery still behaves like SQLite's. Click pages in the demo to corrupt them and see how many losses a given amount of repair data can absorb.
Erasure-Coded Streams
DesignNative mode replaces the mutable database file with an append-only stream of commit capsules. Each capsule holds a commit's page changes and the evidence its validation used, and is stored as RaptorQ symbols. A commit counts once its marker, which points to the previous marker, is durable. A standard .db can still be exported for compatibility.
Because the code is , the original bytes are stored unchanged and ordinary reads don't decode anything. This is design plus partial implementation: native WAL publication, recovery and page groups are landing on main, but there is no stable switch for it yet. Step through below to see a page split into symbols, encoded, and recovered.
Two Storage Modes
PartialCompatibility mode, on standard SQLite files, is what runs today and what everything else on this site assumes. Native mode is where the project is heading. The comparison below follows one write through each to show where they diverge.
Compact Version Storage
Built, not wiredStoring a full page copy for every version gets expensive when most writes change a few bytes. Storing the XOR of two consecutive versions keeps only what changed, and a full snapshot can reset the chain when a page changes a lot. Note that this is about storing versions compactly, which is safe; it is not the same thing as merging two writers' changes byte by byte, which the design forbids.
Ideas in the Tree
Techniques from the database research literature that have implementations in the workspace but don't sit on the query or cache path of the default runtime.
Learned Indexes
Built, not wiredA B-tree finds a key by walking from the root to a leaf, three or four page hops for a typical table. A learned index fits a small piecewise-linear model to the sorted keys and predicts where a key should be, then searches a bounded window around the guess. It works best on large, read-mostly data with a smooth key distribution.
Database Cracking
Built, not wiredCracking builds an index as a side effect of queries. Each range query partitions the column around its bounds, so the data gradually sorts itself along the ranges people actually ask for. Run the three queries below and watch the array reorganize.
The Cooling Protocol
Built, not wiredAn idea borrowed from LeanStore: instead of evicting the least recently used page directly, pages pass through a “cooling” stage first, and a page that gets touched again while cooling goes back to hot. Click pages below to re-heat them, then run a background scan to watch the cycle.
Detecting Workload Shifts
Built, not wiredkeeps a running estimate of how long the current workload “regime” has lasted and notices when the throughput pattern changes, for example from steady OLTP to a bulk load. FrankenSQLite has an implementation in its MVCC crate, but nothing calls it: garbage collection, checkpointing and eviction don't consult it, and the README describes it as advisory. Start the telemetry below and switch regimes to watch the detector respond.
How It Gets Checked
Most of the project's effort goes into checking that the engine behaves like SQLite and that concurrency doesn't corrupt anything. These techniques mostly live in the test harness; two of them also have narrow uses inside the engine, noted below.
Exploring Thread Schedules
Test harnessRandom stress tests only sample the possible orderings of concurrent operations. group orderings that differ only in the order of independent operations, and runs one representative from each group. FrankenSQLite's harness enumerates the orderings of small transaction models and groups them this way (two three-step transactions have 20 orderings but only 2 distinct outcomes), and the asupersync lab runtime's DPOR explorer checks small MVCC models within step limits. Step through below to see orderings collapse into classes.
Anytime-Valid Monitoring
PartialIf you check an invariant after every operation with an ordinary statistical test, you will eventually get a false alarm just by checking so often. are built to be checked continuously while keeping the false-alarm rate under a fixed bound, which makes them a good fit for long soak tests. In FrankenSQLite they show up in three places: the harness uses one to track SSI's false-positive rate; each connection carries a small monitor fed with conflict and cache statistics, which can only cancel work for high-priority contexts and so does nothing by default; and a research-grade opt-in mode (PRAGMA fsqlite.write_merge = LAB_UNSAFE) lets an e-process gate skip some SSI checks, which the default never does. Run the monitor below, then inject a violation.
Gluing Local Views
Test harnessSome concurrency bugs only show up globally: every pair of transactions looks consistent, but there's no single database state all of them could have seen. A treats each transaction's view as a local section and asks whether they glue into one global state. The version in FrankenSQLite's MVCC crate is a model-level test that works pairwise, flagging two snapshots whose shared reads can't be put in one order; it doesn't open real connections. The example below shows that case.
Distribution-Free Bounds
PartialLatencies are skewed and multi-modal, so mean plus or minus a standard deviation misleads. produces intervals that hold without assuming a distribution. The engine uses it in two places. On Linux, the io_uring backend keeps a window of recent I/O latencies and falls back to plain Unix I/O if one is slower than a conformal upper bound. And, only if you ask, PRAGMA fsqlite.retry_slo_ms caps how long busy retries may take, calibrated from recent retries. Using it as a performance release gate is still a design target.
All 28 Crates
Public API: Connection::open, execute, query, prepare
Connection hub: statement dispatch, transactions, schema, VDBE bridge
PageNumber, PageSize, TxnId, SqliteValue, opcodes, serial types
Error variants, SQLite error-code mapping, transient-error detection
OS abstraction: files, locks, mmap and shared memory
Page cache (S3-FIFO eviction), rollback journal, write-back
frames, checkpoints, WAL index, crash recovery
Page-level , snapshots, SSI validation, version GC
B-tree cells, page splits, overflow chains, cursors
Typed AST for statements and expressions
Hand-written lexer and parser, Pratt expressions
Name resolution, WHERE analysis, join ordering, index choice
bytecode VM with 190+ opcodes and code generation
Scalar, aggregate, window, date/time and math functions
FTS5 full-text search with BM25 ranking
FTS3/FTS4 query helpers (no virtual-table module yet)
JSON1 functions plus json_each and json_tree
R-tree spatial indexes and geopoly functions
Changesets and patchsets (manual library API)
ICU collation and Unicode case folding
generate_series plus uuid and decimal helpers
Metrics, tracing, latency and conflict telemetry
The fsqlite shell: REPL, -c, .read, .dump, output modes
Optional SQLite-style C ABI shim
Experimental WebAssembly build behind the TypeScript SDK
Conformance and differential testing against C SQLite
Workload replay, swarm tests and the benchmark matrix
Health checks for Beads issue databases