{"id":30528014,"url":"https://github.com/mehrantsi/FeOxDB","last_synced_at":"2025-08-27T04:03:16.096Z","repository":{"id":311009761,"uuid":"1040762377","full_name":"mehrantsi/FeOxDB","owner":"mehrantsi","description":"FeOx (Iron-Oxide) is an ultra-fast, embedded and persisted KV store in pure Rust.","archived":false,"fork":false,"pushed_at":"2025-08-21T14:57:16.000Z","size":95,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-08-21T16:35:21.920Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mehrantsi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-19T13:21:16.000Z","updated_at":"2025-08-21T15:03:36.000Z","dependencies_parsed_at":"2025-08-21T16:48:27.907Z","dependency_job_id":null,"html_url":"https://github.com/mehrantsi/FeOxDB","commit_stats":null,"previous_names":["mehrantsi/feoxdb"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/mehrantsi/FeOxDB","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mehrantsi%2FFeOxDB","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mehrantsi%2FFeOxDB/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mehrantsi%2FFeOxDB/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mehrantsi%2FFeOxDB/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mehrantsi","download_url":"https://codeload.github.com/mehrantsi/FeOxDB/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mehrantsi%2FFeOxDB/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272288944,"owners_count":24907778,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-27T02:00:09.397Z","response_time":76,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-27T04:02:11.879Z","updated_at":"2025-08-27T04:03:16.075Z","avatar_url":"https://github.com/mehrantsi.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"logo.svg\" alt=\"FeOxDB Logo\" width=\"400\"\u003e\n\n  Ultra-fast, embedded key-value database for Rust with sub-microsecond latency.\n\u003c/div\u003e\n\n[📚 Documentation](https://feoxdb.com) | [📊 Benchmarks](https://feoxdb.com/benchmarks.html) | [💬 Issues](https://github.com/mehrantsi/FeOxDB/issues)\n\n## Features\n\n- **Sub-Microsecond Latency**: \u003c200ns GET, \u003c600ns INSERT operations\n- **Lock-Free Concurrency**: Built on DashMap and Crossbeam SkipList\n- **io_uring Support** (Linux): Kernel-bypass I/O for maximum throughput with minimal syscalls\n- **Flexible Storage**: Memory-only or persistent modes with async I/O\n- **JSON Patch Support**: RFC 6902 compliant partial updates for JSON values\n- **Atomic Counters**: Thread-safe increment/decrement operations\n- **Write Buffering**: Sharded buffers with batched writes to reduce contention\n- **CLOCK Cache**: Second-chance eviction algorithm\n- **Statistics**: Real-time performance monitoring\n- **Free Space Management**: Dual RB-tree structure for O(log n) allocation\n- **Zero Fragmentation**: Automatic coalescing prevents disk fragmentation\n\n## ACID Properties and Durability\n\nFeOxDB provides ACI properties with relaxed durability:\n\n- **Atomicity**: ✅ Individual operations are atomic via Arc-wrapped records\n- **Consistency**: ✅ Timestamp-based conflict resolution ensures consistency\n- **Isolation**: ✅ Lock-free reads and sharded writes provide operation isolation\n- **Durability**: ⚠️ Write-behind logging with bounded data loss window\n\n### Durability Trade-offs\n\nFeOxDB trades full durability for extreme performance:\n- **Write-behind buffering**: Flushes every 100ms or when buffers fill (1024 entries or 16MB per shard)\n- **Worst-case data loss**:\n  - **Time window**: `100ms + 16MB / 4KB_random_write_QD1_throughput`\n  - **Data at risk**: `16MB × num_shards (num_shards = num_cpus / 2)` (e.g., 64MB for 4 shards, 128MB for 8 shards)\n  - Workers write in parallel, so time doesn't multiply with shards\n  - Example (50MB/s 4KB random QD1): 420ms window, up to 64MB at risk (4 shards)\n  - Example (200MB/s 4KB random QD1): 180ms window, up to 64MB at risk (4 shards)\n- **Memory-only mode**: No durability, maximum performance\n- **Explicit flush**: Call `store.flush()` to synchronously write all buffered data (blocks until fsync completes)\n\n### FAQ:\n\nQ: Would the durability tradeoff for extreme performance worth it?\n  - For KV stores, there are more use cases that can accept this _slightly_ relaxed durability model than not. of course this isn't the case for a main DB, but KV stores often handle derived data, caches, or state that can be rebuilt.\n    That said, for cases needing stronger durability, you can call `store.flush()` after critical operations - gives you fsync-level guarantees.\n    The philosophy is: make the fast path really fast for those who need it, but provide escape hatches for stronger guarantees when needed.\n\nQ: What kind of applications would need this performance? Why these latency numbers matter?\n  - The real value isn't just raw speed - it's efficiency.\n    When operations complete in 200ns instead of blocking for microseconds/milliseconds on fsync, you avoid thread pool exhaustion and connection queueing. Each sync operation blocks that thread until disk confirms - tying up memory, connection slots, and causing tail latency spikes.\n\n    With FeOxDB's write-behind approach:\n      - Operations return immediately, threads stay available\n      - Background workers batch writes, amortizing sync costs across many operations\n      - Same hardware can handle 100x more concurrent requests\n      - Lower cloud bills from needing fewer instances\n\n     For desktop apps, this means your KV store doesn't tie up threads that the UI needs. For servers, it means handling more users without scaling up.\n     The durability tradeoff makes sense when you realize most KV workloads are derived data that can be rebuilt. Why block threads and exhaust IOPS for fsync-level durability on data that doesn't need it?\n   \n## Quick Start\n\n### Installation\n\n```toml\n[dependencies]\nfeoxdb = \"0.1.0\"\n```\n\n### Basic Usage\n\n```rust\nuse feoxdb::FeoxStore;\n\nfn main() -\u003e feoxdb::Result\u003c()\u003e {\n    // Create an in-memory store\n    let store = FeoxStore::new(None)?;\n    \n    // Insert a key-value pair\n    store.insert(b\"user:123\", b\"{\\\"name\\\":\\\"Mehran\\\"}\")?;\n    \n    // Get a value\n    let value = store.get(b\"user:123\")?;\n    println!(\"Value: {}\", String::from_utf8_lossy(\u0026value));\n    \n    // Check existence\n    if store.contains_key(b\"user:123\") {\n        println!(\"Key exists!\");\n    }\n    \n    // Delete a key\n    store.delete(b\"user:123\")?;\n    \n    Ok(())\n}\n```\n\n### Persistent Storage\n\n```rust\nuse feoxdb::FeoxStore;\n\nfn main() -\u003e feoxdb::Result\u003c()\u003e {\n    // Create a persistent store\n    let store = FeoxStore::new(Some(\"/path/to/data.feox\".to_string()))?;\n    \n    // Operations are automatically persisted\n    store.insert(b\"config:app\", b\"production\")?;\n    \n    // Flush to disk\n    store.flush();\n    \n    // Data survives restarts\n    drop(store);\n    let store = FeoxStore::new(Some(\"/path/to/data.feox\".to_string()))?;\n    let value = store.get(b\"config:app\")?;\n    assert_eq!(value, b\"production\");\n    \n    Ok(())\n}\n```\n\n### Advanced Configuration\n\n```rust\nuse feoxdb::FeoxStore;\n\nfn main() -\u003e feoxdb::Result\u003c()\u003e {\n    let store = FeoxStore::builder()\n        .device_path(\"/data/myapp.feox\")\n        .file_size(10 * 1024 * 1024 * 1024)  // 10GB initial file size\n        .max_memory(2_000_000_000)  // 2GB limit\n        .enable_caching(true)        // Enable CLOCK cache\n        .hash_bits(20)               // 1M hash buckets\n        .enable_ttl(true)            // Enable TTL support\n        .build()?;\n    \n    Ok(())\n}\n```\n\n### Time-To-Live (TTL) Support\n\n```rust\nuse feoxdb::FeoxStore;\n\n// Enable TTL feature via builder\nlet store = FeoxStore::builder()\n    .enable_ttl(true)\n    .build()?;\n\n// Set key to expire after 60 seconds\nstore.insert_with_ttl(b\"session:123\", b\"session_data\", 60)?;\n\n// Check remaining TTL\nif let Some(ttl) = store.get_ttl(b\"session:123\")? {\n    println!(\"Session expires in {} seconds\", ttl);\n}\n\n// Extend TTL to 120 seconds\nstore.update_ttl(b\"session:123\", 120)?;\n\n// Remove TTL (make permanent)\nstore.persist(b\"session:123\")?;\n```\n\n### Concurrent Access\n\n```rust\nuse feoxdb::FeoxStore;\nuse std::sync::Arc;\nuse std::thread;\n\nfn main() -\u003e feoxdb::Result\u003c()\u003e {\n    let store = Arc::new(FeoxStore::new(None)?);\n    let mut handles = vec![];\n    \n    // Spawn 10 threads, each inserting data\n    for i in 0..10 {\n        let store_clone = Arc::clone(\u0026store);\n        handles.push(thread::spawn(move || {\n            for j in 0..1000 {\n                let key = format!(\"thread_{}:key_{}\", i, j);\n                store_clone.insert(key.as_bytes(), b\"value\", None).unwrap();\n            }\n        }));\n    }\n    \n    for handle in handles {\n        handle.join().unwrap();\n    }\n    \n    println!(\"Total keys: {}\", store.len());  // 10,000\n    Ok(())\n}\n```\n\n### Range Queries\n\n```rust\nuse feoxdb::FeoxStore;\n\nfn main() -\u003e feoxdb::Result\u003c()\u003e {\n    let store = FeoxStore::new(None)?;\n    \n    // Insert sorted keys\n    store.insert(b\"user:001\", b\"Mehran\")?;\n    store.insert(b\"user:002\", b\"Bob\")?;\n    store.insert(b\"user:003\", b\"Charlie\")?;\n    store.insert(b\"user:004\", b\"David\")?;\n    \n    // Range query: get users 001-003 (inclusive on both ends)\n    let results = store.range_query(b\"user:001\", b\"user:003\", 10)?;\n    \n    for (key, value) in results {\n        println!(\"{}: {}\", \n            String::from_utf8_lossy(\u0026key),\n            String::from_utf8_lossy(\u0026value));\n    }\n    // Outputs: user:001, user:002, user:003\n    \n    Ok(())\n}\n```\n\n### JSON Patch Operations (RFC 6902)\n\nFeOxDB supports partial updates to JSON documents using the standard JSON Patch format:\n\n```rust\nuse feoxdb::FeoxStore;\n\nfn main() -\u003e feoxdb::Result\u003c()\u003e {\n    let store = FeoxStore::new(None)?;\n    \n    // Store a JSON document\n    let user = r#\"{\n        \"name\": \"Mehran\",\n        \"age\": 30,\n        \"skills\": [\"Rust\", \"Go\"],\n        \"address\": {\n            \"city\": \"San Francisco\",\n            \"zip\": \"94105\"\n        }\n    }\"#;\n    store.insert(b\"user:123\", user.as_bytes())?;\n    \n    // Apply patches to modify specific fields\n    let patches = r#\"[\n        {\"op\": \"replace\", \"path\": \"/age\", \"value\": 31},\n        {\"op\": \"add\", \"path\": \"/skills/-\", \"value\": \"Python\"},\n        {\"op\": \"add\", \"path\": \"/email\", \"value\": \"mehran@example.com\"},\n        {\"op\": \"replace\", \"path\": \"/address/city\", \"value\": \"Seattle\"}\n    ]\"#;\n    \n    store.json_patch(b\"user:123\", patches.as_bytes())?;\n    \n    // Document is now updated with patches applied\n    let updated = store.get(b\"user:123\")?;\n    println!(\"Updated: {}\", String::from_utf8_lossy(\u0026updated));\n    \n    Ok(())\n}\n```\n\nSupported JSON Patch operations:\n- `add`: Add a new field or array element\n- `remove`: Remove a field or array element\n- `replace`: Replace an existing value\n- `move`: Move a value from one path to another\n- `copy`: Copy a value from one path to another\n- `test`: Test that a value at a path equals a specified value\n\n### Atomic Counter Operations\n\n```rust\nuse feoxdb::FeoxStore;\n\nfn main() -\u003e feoxdb::Result\u003c()\u003e {\n    let store = FeoxStore::new(None)?;\n    \n    // Initialize counters (must be 8-byte i64 values)\n    let zero: i64 = 0;\n    store.insert(b\"stats:visits\", \u0026zero.to_le_bytes())?;\n    store.insert(b\"stats:downloads\", \u0026zero.to_le_bytes())?;\n    \n    // Increment atomically (thread-safe)\n    let visits = store.atomic_increment(b\"stats:visits\", 1)?;\n    println!(\"Visits: {}\", visits);  // 1\n    \n    // Increment by 10\n    let downloads = store.atomic_increment(b\"stats:downloads\", 10)?;\n    println!(\"Downloads: {}\", downloads);  // 10\n    \n    // Decrement\n    let visits = store.atomic_increment(b\"stats:visits\", -1)?;\n    println!(\"Visits after decrement: {}\", visits);  // 0\n    \n    Ok(())\n}\n```\n\n## Performance\n\n### Benchmarks\n\nRun the included benchmarks:\n\n```bash\n# Performance test\ncargo run --release --example performance_test 100000 100\n\n# Deterministic test\ncargo run --release --example deterministic_test 100000 100\n\n# Criterion benchmarks\ncargo bench\n```\n\n### Results\n\nTypical performance on M3 Max:\n\n| Operation | Latency | Throughput | Notes |\n|-----------|---------|------------|-------|\n| GET | ~200-260ns | 2.1M ops/sec | DashMap lookup + stats |\n| INSERT | ~700ns | 850K ops/sec | Memory allocation + indexing |\n| DELETE | ~290ns | 1.1M ops/sec | Removal from indexes |\n| Mixed (80/20 R/W) | ~290ns | 3.1M ops/sec | Real-world workload |\n\n### Throughput\n\nBased on Criterion benchmarks:\n- **GET**: 8.2M - 10.5M ops/sec (varies by batch size)\n- **INSERT**: 730K - 1.1M ops/sec (varies by value size)\n- **Mixed workload (80/20)**: 3.1M ops/sec\n\n## Architecture\n\nFeOxDB uses a lock-free, multi-tier architecture optimized for modern multi-core CPUs:\n\n### Lock-Free Design\n\nThe entire hot path is lock-free, ensuring consistent sub-microsecond latency:\n\n- **DashMap**: Sharded hash table with lock-free reads and fine-grained locking for writes\n- **Crossbeam SkipList**: Fully lock-free ordered index for range queries\n- **Atomic Operations**: All metadata updates use atomic primitives\n- **RCU-style Access**: Read-Copy-Update pattern for zero-cost reads\n\n### Async Write-Behind Logging\n\nWrites are decoupled from disk I/O for maximum throughput:\n\n1. **Sharded Write Buffers**\n   - Multiple buffers with thread-consistent assignment\n   - Reduces contention between threads\n   - Cache-friendly access patterns\n\n2. **Batched Flushing**\n   - Writes accumulated and flushed in batches\n   - Optimal disk utilization with large sequential writes\n   - Configurable flush intervals\n\n3. **io_uring Integration** (Linux)\n   - Kernel-bypass I/O with submission/completion queues\n   - Zero-copy operations where possible\n   - Async I/O without thread pool overhead\n\n4. **Write Coalescing**\n   - Multiple updates to same key automatically merged\n   - Reduces write amplification\n   - Improves SSD lifespan\n\n### Storage Tiers\n\n1. **In-Memory Layer**\n   - Primary storage in DashMap\n   - O(1) lookups with ~100ns latency\n   - Automatic memory management\n\n2. **Write Buffer Layer**\n   - Sharded buffers for concurrent writes\n   - Lock-free MPSC queues\n   - Backpressure handling\n\n3. **Persistent Storage**\n   - Sector-aligned writes (4KB blocks)\n   - Write-ahead logging for durability\n   - Crash recovery support\n\n4. **Cache Layer**\n   - CLOCK eviction algorithm\n   - Keeps hot data in memory after disk write\n   - Transparent cache management\n\n## API Documentation\n\nFull API documentation is available:\n\n```bash\ncargo doc --open\n```\n\nKey types:\n- `FeoxStore` - Main database interface\n- `StoreBuilder` - Configuration builder\n- `FeoxError` - Error types\n- `Statistics` - Performance metrics\n\n## Examples\n\nSee the `examples/` directory for more:\n\n- `performance_test.rs` - Benchmark tool\n- `deterministic_test.rs` - Reproducible performance test\n\n### Contributing\n\nContributions are welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for more information.\n\n## License\n\nCopyright 2025 Mehran Toosi\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n\nSee [LICENSE](LICENSE) for the full license text.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmehrantsi%2FFeOxDB","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmehrantsi%2FFeOxDB","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmehrantsi%2FFeOxDB/lists"}