{"id":33863468,"url":"https://github.com/YUX/axion","last_synced_at":"2025-12-14T10:00:54.849Z","repository":{"id":327168892,"uuid":"1108163781","full_name":"YUX/axion","owner":"YUX","description":null,"archived":false,"fork":false,"pushed_at":"2025-12-03T09:22:11.000Z","size":1304,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-04T23:02:02.900Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://yux.github.io/axion/","language":"Zig","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/YUX.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-02T05:29:58.000Z","updated_at":"2025-12-03T09:22:10.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/YUX/axion","commit_stats":null,"previous_names":["yux/axion"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/YUX/axion","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YUX%2Faxion","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YUX%2Faxion/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YUX%2Faxion/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YUX%2Faxion/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/YUX","download_url":"https://codeload.github.com/YUX/axion/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YUX%2Faxion/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":27725914,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-14T02:00:11.348Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-12-09T11:00:31.923Z","updated_at":"2025-12-14T10:00:54.833Z","avatar_url":"https://github.com/YUX.png","language":"Zig","readme":"![](img/cover.jpeg)\n# Axion ⚡\n\n**The High-Performance, Embeddable Storage Engine for Zig \u0026 SQLite.**\n\nAxion is a next-generation LSM-tree storage engine written in **Zig**. It is designed to exploit the full potential of modern NVMe SSDs and multi-core CPUs, offering distinct advantages: **blazing fast writes**, **ACID compliance**, **MVCC**, and seamless integration as a **SQLite Virtual Table**.\n\nWhether you need a raw key-value store for your Zig application or a turbo-charged backend for your SQLite queries, Axion delivers.\n\n---\n\n## 🚧 Development Notice\n\n\u003e **Warning:** Axion is currently in **heavy development** and relies on features anticipated in the upcoming **Zig 0.16** release. It is **not yet suitable for production environments**.\n\u003e\n\u003e APIs, file formats, and internal structures may change without notice. We welcome intrepid contributors and feedback, but please use with caution!\n\n---\n\n## 🚀 Key Features\n\n*   **LSM-Tree Architecture:** Optimized for heavy write workloads with Leveled Compaction.\n*   **ACID Transactions:** Strict serializability and snapshot isolation using MVCC (Multi-Version Concurrency Control).\n*   **Modern I/O:** Utilizes `io_uring` on Linux for asynchronous, high-throughput WAL writes.\n*   **SQLite Integration:** Drop-in replacement for SQLite tables via the Virtual Table mechanism. Run SQL on top of an LSM tree!\n*   **Efficiency:**\n    *   **Prefix Compression:** Reduces storage footprint for keys with common prefixes.\n    *   **Block Cache:** Sharded, concurrent block cache to minimize disk reads.\n    *   **Bloom Filters:** fast negative lookups to avoid unnecessary disk seeks.\n    *   **Zero-Copy:** Optimized read paths to reduce memory churn.\n*   **Observability:** Built-in structured logging and atomic metrics for deep runtime insights.\n\n---\n\n## ⚡ Quick Start\n\n### As a Zig Library\n\nAdd Axion to your `build.zig.zon` and import it:\n\n```zig\nconst std = @import(\"std\");\nconst Axion = @import(\"axion\").DB;\n\npub fn main() !void {\n    var gpa = std.heap.GeneralPurposeAllocator(.{}){};\n    const allocator = gpa.allocator();\n\n    // Open the database\n    var db = try Axion.open(allocator, \"./my_data\", .{\n        .memtable_size_bytes = 64 * 1024 * 1024,\n        .block_cache_size_bytes = 1 * 1024 * 1024 * 1024, // 1GB Cache\n    });\n    defer db.close();\n\n    // Write\n    try db.put(\"user:101\", \"Alice\");\n\n    // Read\n    if (try db.get(\"user:101\")) |value| {\n        defer value.deinit();\n        std.debug.print(\"Found: {s}\\n\", .{value.data});\n    }\n\n    // Atomic Transaction\n    var txn = try db.beginTransaction();\n    defer txn.deinit();\n    \n    try txn.put(\"balance:A\", \"500\");\n    try txn.put(\"balance:B\", \"300\");\n    try txn.commit();\n}\n```\n\n### As a SQLite Storage Backend\n\nAxion exposes itself as a SQLite Virtual Table module. This allows you to keep the SQL language you love but swap the B-Tree engine for Axion's LSM tree.\n\n```bash\n# Start the Axion Shell (bundled with SQLite)\n./zig-out/bin/axion_shell\n```\n\n```sql\n-- Mount an Axion database as a virtual table\n-- Syntax: CREATE VIRTUAL TABLE \u003cname\u003e USING axion('\u003cpath\u003e', '\u003cwal_mode\u003e', '', '\u003coptions\u003e');\nCREATE VIRTUAL TABLE users USING axion('./users_db', 'NORMAL', '', 'memtable_mb=64;cache_mb=256');\n\n-- Use standard SQL\nINSERT INTO users (key, value) VALUES ('u1', '{\"name\": \"John\", \"age\": 30}');\nINSERT INTO users (key, value) VALUES ('u2', '{\"name\": \"Jane\", \"age\": 25}');\n\nSELECT * FROM users WHERE key \u003e= 'u1';\n```\n\n### Foreign Language Support (C, Python, Rust, Go)\n\nAxion provides a native C ABI (`libaxion`), making it easy to integrate with any language that supports FFI (Foreign Function Interface) or CGO.\n\n**1. Build the Shared Library:**\n```bash\nzig build\n# Output: zig-out/lib/libaxion.so (or .dll/.dylib)\n# Header: include/axion.h\n```\n\n**2. Usage in C:**\n```c\n#include \"axion.h\"\n#include \u003cstdio.h\u003e\n\nint main() {\n    axion_db_t* db;\n    if (axion_db_open(\"./c_data\", \u0026db) != 0) return 1;\n\n    axion_db_put(db, \"key\", 3, \"val\", 3);\n\n    char* val;\n    size_t len;\n    if (axion_db_get(db, \"key\", 3, \u0026val, \u0026len) == 0 \u0026\u0026 val) {\n        printf(\"Got: %.*s\\n\", (int)len, val);\n        axion_db_free_val(val, len);\n    }\n\n    axion_db_close(db);\n    return 0;\n}\n```\n\n**3. Usage in Python, Rust, Go:**\n*   **Python:** Use `ctypes` or `cffi` to load `libaxion.so`.\n*   **Rust:** Use `bindgen` to generate bindings from `axion.h`.\n*   **Go:** Use `cgo` to link against `libaxion`.\n\n---\n\n## 🛠️ Build \u0026 Test\n\nAxion requires **Zig 0.16 (master)**.\n\n```bash\n# Build release optimized binaries\nzig build -Doptimize=ReleaseFast\n\n# Run the full test suite (Unit + Integration)\nzig build test\n\n# Run the benchmark suite\nzig build -Doptimize=ReleaseFast bench_native_axion bench_native_sqlite bench_vtab_axion     \n```\n\n---\n\n## 📐 Architecture\n\nAxion is built on first principles, avoiding over-engineering while ensuring robustness.\n\n```ascii\n+-----------------------------------------------------------------------+\n|                      Axion - Architecture Overview                    |\n+-----------------------------------------------------------------------+\n\n   User Application (Zig / C / Go / Python)    OR    SQLite Query Engine\n          |                                              |\n          | (Native API)                                 | (Virtual Table)\n          v                                              v\n+-----------------------------------------------------------------------+\n|  INTERFACE LAYER:  c_api.zig  /  sqlite/vtab.zig                      |\n+-----------------------------------------------------------------------+\n          |\n          v\n+-----------------------------------------------------------------------+\n|  TRANSACTION LAYER (MVCC)                                             |\n|                                                                       |\n|  [ CommitBatcher ] \u003c---(Group Commit)---\u003e [ Transaction Manager ]     |\n|         |                                          |                  |\n|         +--- Serializes WAL Entries                +-- Global Version |\n|         +--- Detects Conflicts                     +-- Snapshots      |\n+-----------------------------------------------------------------------+\n          | (Writes)                                 | (Reads)\n          v                                          v\n+--------------------------+           +--------------------------------+\n|  STORAGE LAYER (LSM)     |           |  READ PATH                     |\n|                          |           |                                |\n|  +--------------------+  |           |  1. Check MemTable (Skiplist)  |\n|  | MemTable (Memory)  |  |           |     (Latest updates)           |\n|  | [Sharded Skiplist] |  |           |                                |\n|  +--------------------+  |           |             v                  |\n|            | (Flush)     |           |                                |\n|            v             |           |  2. Check Block Cache (LRU)    |\n|  +--------------------+  |           |     (Hot data in memory)       |\n|  | WAL (Disk)         |  |           |                                |\n|  | [Append-Only Log]  |  |           |             v                  |\n|  +--------------------+  |           |                                |\n|            | (Compact)   |           |  3. Check Bloom Filters        |\n|            v             |           |     (Skip missing keys)        |\n|  +--------------------+  |           |                                |\n|  | SSTables (Disk)    |  |           |             v                  |\n|  | Level 0 (Unsorted) |  |           |                                |\n|  | Level 1 (Sorted)   | \u003c------------+  4. Scan SSTables (Disk)       |\n|  | Level 2 (Sorted)   |  |              (Binary search blocks)        |\n|  +--------------------+  |                                            |\n+--------------------------+           +--------------------------------+\n\n**Key pieces**\n- `WAL`: append-only log with CRC; replay trims torn tails, resets offsets, and replays idempotently.\n- `MemTable`: 16-shard skiplist for writes; rotates into immutables when the configured byte budget is hit.\n- `VersionSet`: snapshot container (active + immutable memtables, SSTable metadata, block cache handle) with ref-counting so iterators stay stable.\n- `Compaction`: background threads pick work via level/size scores, rewrite SSTables, and install edits atomically with manifest updates.\n- `BlockCache`: sharded LRU keyed by file id + block offset; ref-counted handles avoid copying block data.\n- `TransactionManager`: MVCC with monotonic versions, group commit batching for WAL/memtable application, and conflict checks against recent history plus latest-visible versions.\n\n+-----------------------------------------------------------------------+\n|                      WORKFLOW: WRITE (PUT)                            |\n+-----------------------------------------------------------------------+\n\n1. User calls db.put(key, val)\n2. Transaction created, assigned Sequence Number.\n3. Written to WAL (Write-Ahead Log) for durability.\n   (Optimized: Pre-serialized, patched with checksum later)\n4. Written to MemTable (In-Memory SkipList).\n5. Commit confirmed to user.\n\n... LATER (Background) ...\n\n6. MemTable Full? -\u003e FLUSH to Disk (Level 0 SSTable).\n7. Level 0 Full?  -\u003e COMPACTION (Merge L0 -\u003e L1).\n\n+-----------------------------------------------------------------------+\n|                      WORKFLOW: READ (GET)                             |\n+-----------------------------------------------------------------------+\n\n1. User calls db.get(key)\n2. Acquire Snapshot (Current Max Sequence Number).\n3. Search MemTable. Found? -\u003e Return (if visible to snapshot).\n4. Search Block Cache. Found? -\u003e Return.\n5. For each Level (L0 -\u003e L1...):\n   a. Check Bloom Filter. (Negative? -\u003e Skip File).\n   b. Read Index Block.\n   c. Read Data Block from Disk.\n   d. Check CRC32 Checksum.\n   e. Search Key in Block. Found? -\u003e Return.\n```\n\n---\n\n## 📊 Configuration\n\nAxion can be configured via the Options string in `CREATE VIRTUAL TABLE` or via the C API.\n\n| Option | Default | Description |\n| :--- | :--- | :--- |\n| `memtable_mb` | 64 | Size of in-memory buffer (MB) before flushing. |\n| `cache_mb` | 10 | Size of the in-memory LRU block cache (MB). |\n| `l0_limit` | 4 | Number of SSTables in Level 0 before compaction triggers. |\n| `l1_mb` | 256 | Target size of Level 1 (MB). Subsequent levels are 10x larger. |\n| `compaction_threads` | 1 | Number of background threads for merging tables. |\n| `wal_sync_mode` | `Full` | `Full` (fsync), `Normal` (write), or `Off` (unsafe). |\n| `config_file` | - | Path to an external configuration file (k=v format). |\n\nExample with multiple options:\n```sql\nCREATE VIRTUAL TABLE logs USING axion('./logs_db', 'NORMAL', '', 'memtable_mb=128;l0_limit=8;compaction_threads=2');\n```\n\n---\n\n## 📈 Benchmarks\n\nAxion demonstrates significant performance advantages over standard SQLite (B-Tree) for write-heavy workloads, while maintaining competitive read speeds.\n\n**Environment:** Linux, **Zig 0.16 (master)** (ReleaseFast), 100 Threads, 50k Keys.\n\n### 1. Sync Mode: `FULL` (Strict Fsync Durability)\n*Safety first: Every write is physically flushed to disk.*\n\n| Metric | SQLite (Native) | Axion (Native) | Axion (VTab) |\n| :--- | :--- | :--- | :--- |\n| **Write (Random)** | 170 | **7,286** | 2,984 |\n| **Write (Sequential)** | 145 | **7,501** | 3,131 |\n| **Read (Point)** | 1,272,793 | **3,247,513** | 2,512,192 |\n| **Range Scan** | **608,382** | 241,331 | 147,694 |\n| **Mix (Read/Write)** | **2,783,051** | 13,743 | 6,286 |\n\n### 2. Sync Mode: `NORMAL` (Buffered/Async Writes)\n*Performance balanced: Writes are buffered to OS cache; `io_uring` used where applicable.*\n\n| Metric | SQLite (Native) | Axion (Native) | Axion (VTab) |\n| :--- | :--- | :--- | :--- |\n| **Write (Random)** | 147 | **414,050** | 311,247 |\n| **Write (Sequential)** | 172 | **443,205** | 323,952 |\n| **Read (Point)** | 1,277,213 | **2,631,517** | 2,326,464 |\n| **Range Scan** | **611,262** | 244,644 | 154,379 |\n| **Mix (Read/Write)** | **2,667,385** | 607,981 | 536,348 |\n\n### 3. Sync Mode: `OFF` (Unsafe/Max Speed)\n*Raw in-memory speed: No disk sync guarantees.*\n\n| Metric | SQLite (Native) | Axion (Native) | Axion (VTab) |\n| :--- | :--- | :--- | :--- |\n| **Write (Random)** | 147 | **356,085** | 317,070 |\n| **Write (Sequential)** | 172 | **409,023** | 329,262 |\n| **Read (Point)** | 1,204,112 | **2,507,509** | 2,325,622 |\n| **Range Scan** | **562,927** | 247,068 | 153,207 |\n| **Mix (Read/Write)** | **2,587,663** | 727,515 | 546,838 |\n\n---\n\n## 🖥️ CLI Tools\n\nAxion comes with a handy shell for management and inspection.\n\n```bash\n# Open a DB and check internal stats\n./zig-out/bin/axion_shell ./my_data --sql \".stats\"\n```\n\nOutput:\n```text\n--- Axion Metrics ---\nPuts: 1250430\nGets: 4021\nIterators: 50\nCompactions: 12\nFlushes: 8\nBlockCache Hits: 8540\nBlockCache Misses: 120\nTxn Success: 1250000\nTxn Conflict: 0\n---------------------\n```\n\n---\n\n## 📜 License\n\nThis project is open-source software.\n\n---\n\n## 🤖 AI Usage Disclosure\n\nThis project's codebase was initially written by human developers and has since evolved through AI-assisted audits and contributions.\n\n*   **Code Origin:** The core logic and initial implementation are human-authored.\n*   **AI Role:** AI tools are used for code auditing, documentation generation, website building, and intelligent auto-completion.\n*   **Code Verification:** AI does **not** write code directly without human oversight. There is no \"vibe coding\" involved; all AI-suggested changes are reviewed.\n*   **Documentation:** All documentation and website content are primarily generated and maintained by AI to ensure clarity and consistency.\n","funding_links":[],"categories":["Data \u0026 Science"],"sub_categories":["Database"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FYUX%2Faxion","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FYUX%2Faxion","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FYUX%2Faxion/lists"}