v0.4.13 internals

Architecture

Architecture

Architecture

How the 28 crates fit together, how concurrent writers actually work, and which of the more ambitious ideas are running today versus still on the bench. Every section is labeled.

LiveOn by default in the current runtimeOpt-inWorks today, but you have to turn it onPartialWorks for some paths or configurations, not allBuilt, not wiredThe code exists and is tested, but the default runtime does not call it yetDesignSpecified in the design docs; implementation is incompleteTest harnessUsed to verify the engine, not part of the database runtime
Architecture_Overview
6VERIFICATIONfsqlite-harnessfsqlite-e2ebeads-doctor5INTEGRATIONfsqlite-corefsqlitefsqlite-clifsqlite-observabilityfsqlite-c-apifsqlite-wasm4EXTENSIONSfsqlite-ext-fts5fsqlite-ext-fts3fsqlite-ext-jsonfsqlite-ext-rtreefsqlite-ext-sessionfsqlite-ext-icufsqlite-ext-misc3SQLfsqlite-astfsqlite-parserfsqlite-plannerfsqlite-vdbefsqlite-func2STORAGEfsqlite-vfsfsqlite-pagerfsqlite-walfsqlite-mvccfsqlite-btree1FOUNDATIONfsqlite-typesfsqlite-error

The Layers

Each layer is one or more crates, and Cargo enforces the direction of dependencies. The parser can't reach into the pager, and the B-tree can't call the planner.

Foundation

fsqlite-typesfsqlite-error
Shared types and errors. Page numbers, transaction IDs and page sizes are distinct , so the compiler refuses to mix them up. Errors map onto SQLite's result codes and know whether they are worth retrying.

Storage

fsqlite-vfsfsqlite-pagerfsqlite-walfsqlite-mvccfsqlite-btree
The VFS talks to the OS. The pager caches pages with S3-FIFO eviction (an policy is implemented but not exposed) and manages the rollback journal. The crate handles frames, checkpoints and the shared WAL index. keeps per-page versions and runs SSI validation. The B-tree crate lays out cells, splits pages and walks cursors.

SQL

fsqlite-astfsqlite-parserfsqlite-plannerfsqlite-vdbefsqlite-func
A hand-written parser produces a typed AST. Code generation turns it into bytecode for a register-based VM with 190+ opcodes. The separate planner crate (join ordering, index selection) is substantial but not yet on the hot path for every query.

Extensions

fsqlite-ext-fts5fsqlite-ext-fts3fsqlite-ext-jsonfsqlite-ext-rtreefsqlite-ext-sessionfsqlite-ext-icufsqlite-ext-misc
FTS5, JSON1, R-tree/geopoly, ICU and generate_series register in the live engine. FTS3/FTS4 are helper code without a virtual-table module, and the session crate is a manual library API.

Integration

fsqlite-corefsqlitefsqlite-clifsqlite-observabilityfsqlite-c-apifsqlite-wasm
fsqlite-core ties everything into a Connection, and fsqlite is the crate you depend on. Around them: the fsqlite shell, metrics and tracing, an optional C ABI shim, and an experimental WebAssembly build.

Verification

fsqlite-harnessfsqlite-e2ebeads-doctor
Most of the testing muscle lives here: differential runs against C SQLite, SQL logic tests, crash and fault injection, multi-process swarm tests, and the benchmark matrix.
Running today

The Live Engine

What runs when you call execute today: the query path through the crates, and the part of FrankenSQLite that is both new and live, many writers in one process with serializable isolation.

From SQL to a Row

Live

One point lookup, followed through the workspace. The parser builds an AST, code generation in fsqlite-core and fsqlite-vdbe picks the access path and emits bytecode, the VM walks the B-tree through the pager and MVCC visibility rules, and the WAL layer decides whether the newest copy of a page is in the log or the main file. Step through it below.

MVCC, Page by Page

Live

Snapshots

Every transaction reads from a , identified by the highest commit sequence number it may see. Writers produce new page versions instead of overwriting ones a reader might be using, so readers never block and never see a half-finished write.

Concurrent writers

Plain BEGIN is promoted to BEGIN CONCURRENT. There is no fixed writer cap; open one connection per thread. A writer locks a page the first time it writes it, and if another transaction holds it, the call fails fast instead of waiting. No waiting means no wait-for cycles, so page locks can't deadlock.

Commit

Statement execution and private page changes overlap freely. Commit takes a short registry guard, runs and validation, writes through the pager, and publishes the result to the commit index. The loser of a same-page race gets SQLITE_BUSY_SNAPSHOT.

Cleaning up versions

Versions older than the oldest snapshot anyone still holds can go. The engine keeps the newest version of each page at or below that horizon, retires the rest, and frees them in batches with epoch-based reclamation once no reader can be looking at them.

Time-travel queries

Partial

SELECT ... FOR SYSTEM_TIME AS OF COMMITSEQ n (or a timestamp) works on :memory: databases, which keep a ring of up to 256 snapshots taken at each commit. File-backed databases return an explicit error rather than current data. A .fsqlite-history sidecar for file-backed history is designed but not built.

The Commit Path

Live

Statement execution and page changes run on each connection's own thread, in parallel. Commit is where they meet: the committing connection takes a short per-database guard, runs first-committer-wins and SSI validation, writes its frames through the pager into the WAL, and publishes the result to the commit index, all before releasing the guard. Holding it across all three steps closes a race where a peer could claim a freshly allocated page before the allocation became visible.

The design also calls for a dedicated that batches commits and sequences fsyncs. On the default path that service is still scaffolding. A feature-gated batching service for native-mode commits has just landed on main, but nothing in the public connection drives it yet. The visualization below models the live path, including validation failures that send a connection back to retry.

The WAL Index

Live

With a write-ahead log, the newest copy of a page might be in the WAL rather than the main file. SQLite keeps a hash table in the -shm shared-memory file that maps page numbers to WAL frames, so a reader can find the right frame without scanning the log.

FrankenSQLite reads and writes that structure in SQLite's own format and honors its reader marks and checkpoint rules. Even so, don't point FrankenSQLite and stock SQLite at the same file at the same time; hand a database from one to the other after a checkpoint. Type a page number below to watch the lookup probe the table.

The Page Cache

Live

A plain LRU cache does badly on databases: one big table scan pushes out the pages everything else keeps using. FrankenSQLite's pager uses S3-FIFO by default, which admits new pages to a small probationary queue so one-off scan pages leave quickly.

An policy is implemented in the pager crate too, but connections always open with S3-FIFO and there is no PRAGMA or option to switch. ARC balances a recency list against a frequency list and keeps “ghost” entries for recently evicted pages to learn which side deserves more room. The demo below shows ARC's four lists at work.

Byte-for-Byte File Format

Live

Compatibility comes down to details like this one. SQLite stores rowids, record header sizes and serial types as variable-length integers: one byte for small values, up to nine for the largest. FrankenSQLite encodes and decodes them exactly the same way, along with the rest of the record and B-tree page format, so files move between it and stock sqlite3 without conversion.

Drag the slider below to see how integers of different sizes map to byte widths.

On the bench

Built, Partial, or Designed

These are the ideas the project is known for. Some have working code that isn't connected to the default runtime yet; some are partly wired; some are designs with pieces landing. Each section says which.

The Safe Merge Ladder

Built, not wired

Page-level MVCC has one built-in weakness. Two transactions that insert different rows into the same leaf page conflict, even though their changes don't overlap in any meaningful way. Today the second committer simply retries. The merge ladder is the planned way to let both commit when that is provably safe.

The code is in fsqlite-mvcc and tested, but the live commit path doesn't call it and the intent log isn't recorded during writes yet. Tracked as bd-3d5y3 and bd-p4dcv.

1

Rebase the intent

Each writer logs what it meant to do at the B-tree level (“insert rowid 42”). On conflict, that log against the winner's page. If the B-tree invariants and constraints still hold, commit.

2

Structured patch

If the two transactions touched different cells (compared by cell key, not by byte range), merge the cell changes, serialize any page-header changes, and re-check the page.

3

Abort and retry

A real conflict. The loser gets SQLITE_BUSY_SNAPSHOT. This is the only rung the live engine uses today.

What's deliberately missing: merging raw byte ranges. Two writes can touch disjoint bytes and still collide, for example when one transaction moves a cell and the other edits the cell's old location. A byte-level merge would silently drop the second update, so the design forbids it for B-tree, overflow, freelist and pointer-map pages.

RaptorQ Repair Symbols

Partial

(RFC 6330) is a fountain code. It splits data into K source symbols and can generate as many as you like; almost any K symbols, plus a small margin, are enough to rebuild the original. The math happens in : a fast peeling decoder handles most symbols and finishes the rest.

In the engine today, file-backed connections encode durable WAL ranges into a -wal-fec sidecar in the background after each WAL sync (PRAGMA raptorq_repair_symbols, default 2, sets the budget). On Unix, an explicit repair_and_open call can rebuild damaged frames from it and write a verified backup. Ordinary opens don't do that yet, so default recovery still behaves like SQLite's. Click pages in the demo to corrupt them and see how many losses a given amount of repair data can absorb.

Erasure-Coded Streams

Design

Native mode replaces the mutable database file with an append-only stream of commit capsules. Each capsule holds a commit's page changes and the evidence its validation used, and is stored as RaptorQ symbols. A commit counts once its marker, which points to the previous marker, is durable. A standard .db can still be exported for compatibility.

Because the code is , the original bytes are stored unchanged and ordinary reads don't decode anything. This is design plus partial implementation: native WAL publication, recovery and page groups are landing on main, but there is no stable switch for it yet. Step through below to see a page split into symbols, encoded, and recovered.

Two Storage Modes

Partial

Compatibility mode, on standard SQLite files, is what runs today and what everything else on this site assumes. Native mode is where the project is heading. The comparison below follows one write through each to show where they diverge.

Compact Version Storage

Built, not wired

Storing a full page copy for every version gets expensive when most writes change a few bytes. Storing the XOR of two consecutive versions keeps only what changed, and a full snapshot can reset the chain when a page changes a lot. Note that this is about storing versions compactly, which is safe; it is not the same thing as merging two writers' changes byte by byte, which the design forbids.

Research

Ideas in the Tree

Techniques from the database research literature that have implementations in the workspace but don't sit on the query or cache path of the default runtime.

Learned Indexes

Built, not wired

A B-tree finds a key by walking from the root to a leaf, three or four page hops for a typical table. A learned index fits a small piecewise-linear model to the sorted keys and predicts where a key should be, then searches a bounded window around the guess. It works best on large, read-mostly data with a smooth key distribution.

Database Cracking

Built, not wired

Cracking builds an index as a side effect of queries. Each range query partitions the column around its bounds, so the data gradually sorts itself along the ranges people actually ask for. Run the three queries below and watch the array reorganize.

The Cooling Protocol

Built, not wired

An idea borrowed from LeanStore: instead of evicting the least recently used page directly, pages pass through a “cooling” stage first, and a page that gets touched again while cooling goes back to hot. Click pages below to re-heat them, then run a background scan to watch the cycle.

Detecting Workload Shifts

Built, not wired

keeps a running estimate of how long the current workload “regime” has lasted and notices when the throughput pattern changes, for example from steady OLTP to a bulk load. FrankenSQLite has an implementation in its MVCC crate, but nothing calls it: garbage collection, checkpointing and eviction don't consult it, and the README describes it as advisory. Start the telemetry below and switch regimes to watch the detector respond.

Verification

How It Gets Checked

Most of the project's effort goes into checking that the engine behaves like SQLite and that concurrency doesn't corrupt anything. These techniques mostly live in the test harness; two of them also have narrow uses inside the engine, noted below.

Exploring Thread Schedules

Test harness

Random stress tests only sample the possible orderings of concurrent operations. group orderings that differ only in the order of independent operations, and runs one representative from each group. FrankenSQLite's harness enumerates the orderings of small transaction models and groups them this way (two three-step transactions have 20 orderings but only 2 distinct outcomes), and the asupersync lab runtime's DPOR explorer checks small MVCC models within step limits. Step through below to see orderings collapse into classes.

Anytime-Valid Monitoring

Partial

If you check an invariant after every operation with an ordinary statistical test, you will eventually get a false alarm just by checking so often. are built to be checked continuously while keeping the false-alarm rate under a fixed bound, which makes them a good fit for long soak tests. In FrankenSQLite they show up in three places: the harness uses one to track SSI's false-positive rate; each connection carries a small monitor fed with conflict and cache statistics, which can only cancel work for high-priority contexts and so does nothing by default; and a research-grade opt-in mode (PRAGMA fsqlite.write_merge = LAB_UNSAFE) lets an e-process gate skip some SSI checks, which the default never does. Run the monitor below, then inject a violation.

Gluing Local Views

Test harness

Some concurrency bugs only show up globally: every pair of transactions looks consistent, but there's no single database state all of them could have seen. A treats each transaction's view as a local section and asks whether they glue into one global state. The version in FrankenSQLite's MVCC crate is a model-level test that works pairwise, flagging two snapshots whose shared reads can't be put in one order; it doesn't open real connections. The example below shows that case.

Distribution-Free Bounds

Partial

Latencies are skewed and multi-modal, so mean plus or minus a standard deviation misleads. produces intervals that hold without assuming a distribution. The engine uses it in two places. On Linux, the io_uring backend keeps a window of recent I/O latencies and falls back to plain Unix I/O if one is slower than a conformal upper bound. And, only if you ask, PRAGMA fsqlite.retry_slo_ms caps how long busy retries may take, calibrated from recent retries. Using it as a performance release gate is still a design target.

All 28 Crates

fsqlite

Public API: Connection::open, execute, query, prepare

fsqlite-core

Connection hub: statement dispatch, transactions, schema, VDBE bridge

fsqlite-types

PageNumber, PageSize, TxnId, SqliteValue, opcodes, serial types

fsqlite-error

Error variants, SQLite error-code mapping, transient-error detection

fsqlite-vfs

OS abstraction: files, locks, mmap and shared memory

fsqlite-pager

Page cache (S3-FIFO eviction), rollback journal, write-back

fsqlite-wal

frames, checkpoints, WAL index, crash recovery

fsqlite-mvcc

Page-level , snapshots, SSI validation, version GC

fsqlite-btree

B-tree cells, page splits, overflow chains, cursors

fsqlite-ast

Typed AST for statements and expressions

fsqlite-parser

Hand-written lexer and parser, Pratt expressions

fsqlite-planner

Name resolution, WHERE analysis, join ordering, index choice

fsqlite-vdbe

bytecode VM with 190+ opcodes and code generation

fsqlite-func

Scalar, aggregate, window, date/time and math functions

fsqlite-ext-fts5

FTS5 full-text search with BM25 ranking

fsqlite-ext-fts3

FTS3/FTS4 query helpers (no virtual-table module yet)

fsqlite-ext-json

JSON1 functions plus json_each and json_tree

fsqlite-ext-rtree

R-tree spatial indexes and geopoly functions

fsqlite-ext-session

Changesets and patchsets (manual library API)

fsqlite-ext-icu

ICU collation and Unicode case folding

fsqlite-ext-misc

generate_series plus uuid and decimal helpers

fsqlite-observability

Metrics, tracing, latency and conflict telemetry

fsqlite-cli

The fsqlite shell: REPL, -c, .read, .dump, output modes

fsqlite-c-api

Optional SQLite-style C ABI shim

fsqlite-wasm

Experimental WebAssembly build behind the TypeScript SDK

fsqlite-harness

Conformance and differential testing against C SQLite

fsqlite-e2e

Workload replay, swarm tests and the benchmark matrix

beads-doctor

Health checks for Beads issue databases