{"id":51672528,"url":"https://github.com/dj258255/db-hobby","last_synced_at":"2026-07-15T02:03:45.703Z","repository":{"id":366815080,"uuid":"1277973042","full_name":"dj258255/db-hobby","owner":"dj258255","description":"A relational database built from scratch in C to dissect how PostgreSQL/MySQL work — pages, buffer pool, heap, B+Tree, SQL parser/executor, WAL, transactions.","archived":false,"fork":false,"pushed_at":"2026-07-02T10:55:28.000Z","size":938,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-02T12:18:54.963Z","etag":null,"topics":["b-tree","c","database","database-internals","dbms","from-scratch","learning-project","sql","storage-engine","wal"],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dj258255.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"ROADMAP.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-23T10:53:58.000Z","updated_at":"2026-07-02T10:55:31.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/dj258255/db-hobby","commit_stats":null,"previous_names":["dj258255/minidb","dj258255/db-hobby"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dj258255/db-hobby","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dj258255%2Fdb-hobby","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dj258255%2Fdb-hobby/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dj258255%2Fdb-hobby/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dj258255%2Fdb-hobby/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dj258255","download_url":"https://codeload.github.com/dj258255/db-hobby/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dj258255%2Fdb-hobby/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35486967,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-15T02:00:06.706Z","response_time":131,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["b-tree","c","database","database-internals","dbms","from-scratch","learning-project","sql","storage-engine","wal"],"created_at":"2026-07-15T02:03:44.810Z","updated_at":"2026-07-15T02:03:45.693Z","avatar_url":"https://github.com/dj258255.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# db-hobby\n\nA small relational database written from scratch in C — one that **an actual\n`psql` connects to**, with **MVCC snapshot isolation where readers don't block\nwriters**, WAL crash recovery, VACUUM, a thread-safe buffer pool, and a\ncost-based query planner. And past the single node into the hard parts:\n**primary→replica replication over a real socket, Raft consensus** (leader\nelection, log replication, disk-persisted votes, snapshots, membership changes)\nthat **replicates the real engine into a highly-available DB surviving leader\ndeath**, and an **LSM storage engine**. Built from raw fixed-size pages up to running\nSQL — and out to a small distributed system — to dissect how PostgreSQL and\nMySQL actually work inside.\n\n![db-hobby: two psql clients, readers don't block writers](docs/psql-demo.svg)\n\nTwo real `psql` clients above, interleaving over the PostgreSQL wire protocol:\nwhile A holds an uncommitted `UPDATE`, B reads the **old** version (not blocked),\na second writer is rejected (first-updater-wins), and after A commits B sees the\nnew value. That's MVCC — and it's a 400-line server speaking Postgres's protocol,\nso psql can't tell the difference.\n\nIt's a learning project: the goal isn't to invent something new, it's to\nreproduce the real structure accurately and understand it. Every layer is\ncovered by tests (**694 checks across 42 suites**), and the concurrency is\nverified under ThreadSanitizer. The 11-part deep-dive series (DB Internals via a Mini DB) is at\n[IT-Oasis / db-hobby](https://dj258255.github.io/IT-Oasis/blog/project/db-hobby/db-internals-01-storage).\n\n**What's in it:** page storage · buffer pool (thread-safe, pin protocol) · heap ·\nB+Tree (+ concurrent latch-crabbing variant) · hand-written SQL parser \u0026 executor ·\njoins (nested-loop / index / hash) · aggregates · WAL with **steal + no-force**\nrecovery · **MVCC** (xmin/xmax, snapshot isolation, VACUUM) · multi-transaction\nsessions · a **PostgreSQL-wire server** · a **cost-based optimizer** (ANALYZE,\nselectivity, Selinger join-order DP) · **log-shipping + TCP replication** ·\n**Raft consensus** (election, log replication, persistence, snapshots) · an\n**LSM storage engine**. Heap (PG) vs clustered (InnoDB) storage is benchmarked\nside by side.\n\n![db-hobby REPL demo](docs/demo.svg)\n\n## Quick start\n\n```sh\nmake test            # build and run the test suite\nmake bench           # build (-O2) and run the benchmark (index vs scan, fsync cost)\nmake repl            # build the REPL\n./build/db-hobby my.db # open (or create) a database and type SQL\n\n# or speak the real PostgreSQL wire protocol and connect with psql:\n./build/db-hobby my.db --serve 5433\npsql \"host=127.0.0.1 port=5433 dbname=db-hobby\"\n```\n\nThe `--serve` mode is a **thread-per-connection** server that speaks PostgreSQL's\nv3 wire protocol, so **an actual `psql` connects to it** (tested against psql\n14.19). Each connection gets its own OS thread and one of the multi-transaction\nsessions, so two `psql` windows demonstrate \"readers don't block writers\" over\nthe network. The buffer pool is thread-safe (a per-pool latch; the pin protocol\nprotects the returned page data), verified by a pthreads stress test that runs\nclean under ThreadSanitizer. Execution itself is still serialized by one coarse\nengine latch -- finer-grained latching (B+Tree latch crabbing, a blocking lock\nmanager, dropping the engine latch) is the remaining frontier. (Simple query\nonly -- no extended/prepared protocol; SELECT rows are parsed from the executor's\ntext output, so a `TEXT` value containing `\" | \"` splits columns.)\n\nA session:\n\n```\ndb-hobby\u003e CREATE TABLE users (id INT, name TEXT);\n테이블 'users' 생성됨 (컬럼 2개)\n  (인덱스: id 컬럼)\ndb-hobby\u003e INSERT INTO users VALUES (1, 'kim');\ndb-hobby\u003e INSERT INTO users VALUES (2, 'lee');\ndb-hobby\u003e SELECT * FROM users WHERE id = 2;\nid | name\n2 | lee\n(1행, 인덱스 사용)\ndb-hobby\u003e BEGIN;\ndb-hobby\u003e DELETE FROM users WHERE id = 1;\ndb-hobby\u003e ROLLBACK;\ndb-hobby\u003e SELECT * FROM users;\nid | name\n1 | kim\n2 | lee\n(2행)\n```\n\nEach table is stored as its own pair of files and survives a restart (the schema\nis persisted too, so no need to re-run `CREATE TABLE`) -- see the storage layout\nbelow.\n\n## What's inside\n\nBuilt bottom-up; each layer sits on the one below it.\n\n| Layer | What it does | Mirrors |\n|---|---|---|\n| `pager.c` | fixed-size 4KB pages \u003c-\u003e a single file (`page_id * PAGE_SIZE`) | SQLite pager, PG smgr |\n| `page.c` | slotted page: pack variable-length rows into a page | PG/InnoDB page layout |\n| `bufpool.c` | page cache with pin counts, dirty flags, LRU eviction | InnoDB buffer pool |\n| `heap.c` | table = a collection of pages; rows addressed by `RID = (page, slot)` | PG heap |\n| `sql.c` | hand-written lexer + recursive-descent parser (SQL -\u003e AST) | every DB frontend |\n| `db.c` | executor: tuple codec, multi-table catalog, joins (NLJ/index/hash), aggregates | pg_catalog, executor |\n| `btree.c` | on-disk B+Tree index for O(log n) lookups, with node splits | InnoDB clustered index |\n| `wal.c` | write-ahead log on each table's data file: atomic commit + crash recovery | PG WAL / redo log |\n| `mvcc.c` | transaction-state log + visibility rule (xmin committed AND xmax not) | PG MVCC / tqual |\n| `lock.c` | 2PL table locks (S/X) + deadlock detection (wait-for graph) for isolation | PG/InnoDB lock manager |\n\n### Storage layout\n\nLike PostgreSQL (each relation is its own file, `relfilenode`), every table lives\nin its own files, and a catalog file lists which tables exist:\n\n```\nmydb              catalog -- table names + schemas (like pg_class)\nmydb.users.tbl    users rows  (heap)\nmydb.users.wal    write-ahead log for the heap (commit atomicity + crash recovery)\nmydb.users.idx    users PK index (B+Tree)\nmydb.users.idx.wal  write-ahead log for the index\nmydb.orders.tbl   orders rows\nmydb.orders.wal\nmydb.orders.idx   orders PK index\nmydb.orders.idx.wal\n```\n\n## Beyond the single-node engine\n\nThe core above is one coherent single-node database. On top of it, the harder\naxes of a real system are built as **focused, independently-tested modules** —\neach with an honest boundary spelled out in its part of the build log. Most are\nkept as standalone modules (so the 600+ green tests stay safe), each marking\nwhere it ends and integration would begin — **except four capstones wired\nend-to-end**: WAL replication (a replica replays the *real engine's* committed\nWAL and serves `SELECT`), Raft state-machine replication (`raftdb.c`), the\n**LSM tree as a pluggable PK index** (`CREATE TABLE … USING lsm`), and the\n**parallel scan wired into a real `SELECT`** (workers judge visibility+WHERE,\nthe leader prints — byte-identical to serial, ThreadSanitizer-clean).\n\n| Module | What it does | Mirrors |\n|---|---|---|\n| `raft.c` | **Raft consensus** — leader election, log replication, the five §5 safety properties, disk-persisted term/vote (prevents double-voting across a crash), log compaction + InstallSnapshot (§7), and single-server membership changes (§6). Verified on a *deterministic simulated network* that injects partitions, crashes, and reordering | etcd / Consul Raft |\n| `raftdb.c` | **Raft-replicated HA database** — state-machine replication wiring `raft.c` onto the real `db.c` engine: a write is proposed to Raft, and every node applies the committed command to its own engine. Survives leader death (failover); engines stay consistent; linearizable reads via ReadIndex (a partitioned old leader is refused, not served stale) | etcd / TiKV, replicated SQL |\n| `replica.c` + `replnet.c` | **WAL replication** — a replica tails the primary's WAL and replays committed records (the crash-recovery redo, run as a stream); carried over a real socket via a walsender/walreceiver. **Wired end-to-end**: replicates the real engine's writes into a replica that serves `SELECT` (base snapshot + WAL replay, pg_basebackup-style) | PostgreSQL streaming replication |\n| `lsm.c` | an **LSM-tree** storage engine — memtable → SSTable flush → compaction, tombstone deletes — the write-optimized counterpart to the B+Tree. **Wired into the engine** as a pluggable PK index (`CREATE TABLE … USING lsm`): a multi-value mode holds the non-unique PK→RID multimap that MVCC needs, routed through a small Table Access Method (`pidx_*`) | RocksDB / MyRocks |\n| `joinopt.c` | a **Selinger join-order optimizer** — subset DP (2ⁿ instead of n!), cross-product avoidance, cardinality estimation | System R planner |\n| `cbtree.c` | a **concurrent B+Tree** with latch crabbing (per-node rwlocks), ThreadSanitizer-clean | InnoDB index concurrency |\n| `parscan.c` | a **parallel sequential scan** — worker threads sweep disjoint page ranges over the thread-safe buffer pool, leader merges in page order; identical to serial down to RID order, ThreadSanitizer-clean. **Wired into `exec_select`**: a large-table, subquery-free streaming full-scan `SELECT` (and aggregates / GROUP BY) run the visibility gate + WHERE in parallel; the parallel aggregate path also fixes a silent materialization-cap truncation bug. The first foothold to peel the coarse engine latch off layer by layer | PostgreSQL parallel query |\n\n## SQL supported\n\n```\nCREATE TABLE \u003ct\u003e (\u003ccol\u003e INT|TEXT [NOT NULL], ...)\nCREATE INDEX \u003cname\u003e ON \u003ct\u003e (\u003ccol\u003e)   -- secondary index on an INT column\nINSERT INTO \u003ct\u003e VALUES (\u003cint|'text'\u003e, ...)\nSELECT [DISTINCT] \u003c* | item, ...\u003e\n       FROM \u003ct\u003e [\u003calias\u003e] [[LEFT] JOIN \u003ct2\u003e [\u003calias\u003e] ON \u003ccolref\u003e = \u003ccolref\u003e]...\n                  [WHERE \u003ccond\u003e [AND \u003ccond\u003e] [OR ...]]\n                  [GROUP BY \u003ccol\u003e] [HAVING \u003citem\u003e \u003cop\u003e \u003cvalue\u003e]\n                  [ORDER BY \u003ccolref | position\u003e [ASC|DESC], ...] [LIMIT \u003cn\u003e] [OFFSET \u003cn\u003e]\nUPDATE \u003ct\u003e SET \u003ccol\u003e = \u003cvalue\u003e [WHERE ...]\nDELETE FROM \u003ct\u003e [WHERE ...]\nBEGIN | COMMIT | ROLLBACK\nEXPLAIN \u003cselect\u003e          -- print the query plan instead of running it\n\n\u003citem\u003e   is  \u003ccol\u003e | COUNT(*) | COUNT|SUM|MIN|MAX|AVG(\u003ccol\u003e)\n\u003ccond\u003e   is  \u003ccolref\u003e \u003cop\u003e \u003cvalue\u003e  |  \u003ccolref\u003e \u003cop\u003e (SELECT \u003ccol\u003e FROM \u003ct\u003e [WHERE ...])\n                                   |  \u003ccolref\u003e IS [NOT] NULL\n                                   |  \u003ccolref\u003e [NOT] IN (\u003cvalue\u003e, ...)\n                                   |  \u003ccolref\u003e [NOT] IN (SELECT \u003ccol\u003e FROM \u003ct\u003e [WHERE ...])\n                                   |  \u003ccolref\u003e [NOT] BETWEEN \u003cvalue\u003e AND \u003cvalue\u003e\n                                   |  \u003ccolref\u003e [NOT] LIKE '\u003cpattern\u003e'   (% = any run, _ = one char)\n\u003cop\u003e     is one of  =  !=  \u003c  \u003e  \u003c=  \u003e=\n\u003ccolref\u003e is  [\u003ctable\u003e.]\u003ccol\u003e\n```\n\nAn `=`, `\u003c`, `\u003e`, `\u003c=`, or `\u003e=` on the first column (an `INT` primary key) uses\nthe B+Tree index -- `=` is an O(log n) point lookup, the others walk the linked\nleaf chain as a range scan. A `CREATE INDEX` on a non-PK `INT` column adds a\nsecondary (non-unique) B+Tree; an `=` filter on that column does an index scan\n(`btree_find_all` collects candidate RIDs, then each is heap-fetched and the\n`WHERE` is rechecked to drop deleted/stale rows), and `EXPLAIN` shows `Index Scan\nusing \u003cname\u003e`. `!=`, other conditions, and compound `AND` conditions fall back to\na full scan -- the kind of choice a query planner makes. `ORDER BY`/`LIMIT` and `GROUP BY`/aggregates take a materialize path\n(collect, then sort / sort-group); grouped results can be filtered with `HAVING`\nand ordered by an output column or position (so `ORDER BY 2 DESC` gives top-N by\nan aggregate). `JOIN` is a recursive N-way join that picks a\nmethod per level like an optimizer: index nested-loop (`btree_search`) when the\ninner's primary key is the `ON` key, hash join (build a hash on the inner's join\ncolumn, then O(1) probe) for any other equi-join, else a plain nested-loop scan.\n`LEFT JOIN` preserves unmatched left rows by filling the right side with `NULL`.\n`NULL` can also be stored: `INSERT ... VALUES (1, NULL)` keeps it via a null bitmap\nat the front of each row (the first/PK column stays `NOT NULL`). Either way `COUNT(*)`\ncounts those rows but `COUNT(col)`/`SUM`/`AVG` skip the `NULL`s, and `IS [NOT] NULL` tests\nfor them (`LEFT JOIN ... WHERE right.col IS NULL` is the anti-join). `SELECT\nDISTINCT` dedupes output rows. `IN (1, 2, 3)` tests membership against a literal\nvalue set; `IN (SELECT ...)` runs an uncorrelated subquery once into a value set,\nthen tests membership the same way. `BETWEEN a AND b` is desugared at\nparse time into `\u003e= a AND \u003c= b` (inclusive); `LIKE`/`NOT LIKE` match `%` (any run)\nand `_` (one char) with a backtracking matcher -- both run as a full scan, not via\nthe index. Writes go through a **write-ahead\nlog**: a commit (explicit or per-statement autocommit) stages the transaction's\ndirty pages -- for both the heap and the B+Tree index -- and logs them with a\ncommit marker + `fsync`. That log `fsync` is the **only durability point\n(no-force)**: pages are then written back to the data file without `fsync`, and\nthe log is *not* truncated -- it accumulates committed history as the source of\ntruth, trimmed by a size-threshold checkpoint (and on every reopen). If a\ntransaction's dirty pages outgrow the buffer pool, they are **stolen** (evicted to\ndisk before commit) only after their *before-image* is logged first, so the change\nstays undoable. Crash recovery on reopen walks the log: **redo** each committed\nsegment's after-images in commit order, then **undo** the uncommitted tail's\nbefore-images and truncate any pages it allocated. Rollback undoes the same way.\n\nSee `DESIGN.md` for the full layer map and build order.\n\n## Scope (honest limitations)\n\nKept simple on purpose: the first column of each table is treated as a unique\ninteger primary key; `WHERE` is in disjunctive normal form (AND-groups joined by\nOR, no parentheses); joins are INNER only, each `ON` is a single `=`, chained up\nto 4 tables (`INNER` and `LEFT`, aliases supported, so self-joins work);\nprojection, aggregation, `GROUP BY`, and `HAVING` work over a single table or a\njoin result; `NULL` can be stored (nullable columns) or arise from `LEFT JOIN`, except the PK column; subqueries are\nuncorrelated and single-table/single-column; execution is single-threaded, but\n**multiple transactions interleave** through sessions (`SESSION n` switches the\ncurrent one, up to 8): **readers take no locks** -- a reader is never blocked by\na writer, it simply sees the old version through MVCC visibility, and an open\ntransaction reads from its **begin-time snapshot** (repeatable-read-style; plain\nstatements outside a transaction are read-committed). Writers take table-level\n`X` locks (strict 2PL, held to commit), so a second writer on the same table is\nrejected immediately -- **first-updater-wins at table granularity**. That same\nX lock guarantees one writer per table, which is exactly why the per-table WAL\nrecovery (steal/undo/no-force) still holds unchanged. The PK index is\n**multi-version** (one entry per row version, like PostgreSQL); lookups pick the\nvisible version. Closing the database rolls back any open transactions, like a\nreal server dropping connections. The WAL\nprotects both the data (`.tbl`) and index\n(`.idx`) files; a transaction larger than the buffer pool is handled by stealing\ndirty pages to disk with undo (before-image) logging, and commit is **no-force**\n(one log `fsync`; the log is the source of truth, trimmed by a simple size-threshold\ncheckpoint). Logging is whole-page physical -- redo *and* undo are idempotent, so\npageLSN and CLRs are unnecessary here (a crash-injection test proves recovery\nconverges even when re-crashed mid-undo); fuzzy checkpoints and 3-pass ARIES\nawait the preconditions that actually demand them (concurrency, physiological\nlogging). `DELETE` is MVCC-style: rows are never physically removed -- `DELETE`\n(and `UPDATE`'s old version) just stamps `xmax`, and **every** read path (full\nscan, PK point/range lookup, secondary index, joins, aggregates, subqueries)\nfilters by visibility; dead versions accumulate until **`VACUUM [table]`**\nreclaims them: it deletes their index entries (lazy B+Tree leaf deletion,\nPostgreSQL-nbtree-style -- no merge/redistribute), empties their heap slots,\ncompacts each page (RIDs stay stable), and truncates trailing all-empty pages\n(PG-style conditional truncation). VACUUM refuses to run inside a transaction\n(as in PostgreSQL), and its heap/index cleanup commits atomically through the\nWAL. These are noted in the code where they matter.\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdj258255%2Fdb-hobby","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdj258255%2Fdb-hobby","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdj258255%2Fdb-hobby/lists"}