{"id":15060967,"url":"https://github.com/stereodb/stereodb","last_synced_at":"2025-04-04T10:08:44.968Z","repository":{"id":191167807,"uuid":"590507334","full_name":"StereoDB/StereoDB","owner":"StereoDB","description":"Ultrafast and lightweight in-process memory database written in F# that supports: transactions, secondary indexes, persistence, and data size larger than RAM.","archived":false,"fork":false,"pushed_at":"2025-02-06T14:51:01.000Z","size":1002,"stargazers_count":213,"open_issues_count":18,"forks_count":5,"subscribers_count":7,"default_branch":"dev","last_synced_at":"2025-04-04T10:08:38.068Z","etag":null,"topics":["caching","csharp","database","dotnet","dotnet-core","fsharp","hacktoberfest","in-memory-database"],"latest_commit_sha":null,"homepage":"","language":"F#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/StereoDB.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-18T15:19:57.000Z","updated_at":"2025-03-21T12:34:31.000Z","dependencies_parsed_at":"2023-11-13T21:28:46.556Z","dependency_job_id":"f9e0f0f4-f82b-433c-90d8-55af30549341","html_url":"https://github.com/StereoDB/StereoDB","commit_stats":{"total_commits":39,"total_committers":3,"mean_commits":13.0,"dds":"0.28205128205128205","last_synced_commit":"f6b8e1c2633d4721bfcb751f89f6e9e2a9f7918b"},"previous_names":["stereodb/stereodb"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StereoDB%2FStereoDB","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StereoDB%2FStereoDB/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StereoDB%2FStereoDB/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StereoDB%2FStereoDB/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/StereoDB","download_url":"https://codeload.github.com/StereoDB/StereoDB/tar.gz/refs/heads/dev","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247157283,"owners_count":20893220,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["caching","csharp","database","dotnet","dotnet-core","fsharp","hacktoberfest","in-memory-database"],"created_at":"2024-09-24T23:07:27.219Z","updated_at":"2025-04-04T10:08:44.949Z","avatar_url":"https://github.com/StereoDB.png","language":"F#","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/StereoDB/StereoDB/blob/dev/assets/stereo_db_logo.png\" alt=\"StereoDB logo\" width=\"600px\"\u003e\n\u003c/p\u003e\n\n[![build](https://github.com/StereoDB/StereoDB/actions/workflows/build.yml/badge.svg)](https://github.com/StereoDB/StereoDB/actions/workflows/build.yml)\n[![NuGet](https://img.shields.io/nuget/v/stereodb.svg)](https://www.nuget.org/packages/stereodb/)\n\n#### StereoDB\nUltrafast and lightweight in-process memory database written in F# that supports: transactions, secondary indexes, persistence, and data size larger than RAM. The primary use case for this database is building Stateful Services (API or ETL Worker) that keep all data in memory and can provide millions of RPS from a single node. \n\nSupported features:\n- [x] C# and F# API\n- [x] Basic SQL support\n- [x] Transactions (read-only, read-write)\n- [x] Secondary Indexes\n  - [x] Value Index\n  - [x] MultiValue Index\n  - [x] Range Scan Index\n- [ ] Data size larger than RAM\n- [x] Local file data persistence\n- [ ] Distributed mode\n  - [ ] Server and client discovery\n  - [ ] Range-based sharding\n\n#### Intro to Stateful Services\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/StereoDB/StereoDB/blob/dev/assets/architecture.png\" alt=\"StereoDB logo\" width=\"600px\"\u003e\n\u003c/p\u003e\n\n#### Benchmarks\n\nPure KV workload benchmark (in-process only, without persistence). In [this benchmark](https://github.com/StereoDB/StereoDB/blob/dev/benchmarks/StereoDB.Benchmarks/Benchmarks/StereoDbBenchmark.cs), **we run concurrently 3 million random reads and 100K random writes in 892 ms**.\n\n```\nBenchmarkDotNet=v0.13.5, OS=Windows 11 (10.0.22621.2134/22H2/2022Update/SunValley2)\nAMD Ryzen 7 5800H with Radeon Graphics, 1 CPU, 16 logical and 8 physical cores\n.NET SDK=7.0.400\n  [Host]     : .NET 7.0.10 (7.0.1023.36312), X64 RyuJIT AVX2\n  DefaultJob : .NET 7.0.10 (7.0.1023.36312), X64 RyuJIT AVX2\n\n|    Method | ReadThreadCount | WriteThreadCount | UsersCount | DbReadCount | DbWriteCount |     Mean |    Error |   StdDev | Allocated |\n|---------- |---------------- |----------------- |----------- |------------ |------------- |---------:|---------:|---------:|----------:|\n| ReadWrite |              30 |               30 |    4000000 |     3000000 |       100000 | 891.9 ms | 17.75 ms | 35.86 ms |  13.12 KB |\n```\n\n#### C# API\n```csharp\nusing System;\nusing StereoDB;\nusing StereoDB.CSharp;\n\npublic record Book\n{\n    public int Id { get; init; }\n    public string Title { get; init; }\n    public int Quantity { get; init; }\n}\n\npublic record Order\n{\n    public Guid Id { get; init; }\n    public int BookId { get; init; }\n    public int Quantity { get; init; }\n}\n\npublic class BooksSchema\n{\n    public ITable\u003cint, Book\u003e Table { get; init; }\n}\n\npublic class OrdersSchema\n{\n    public ITable\u003cGuid, Order\u003e Table { get; init; }\n    public IValueIndex\u003cint, Order\u003e BookIdIndex { get; init; }\n}\n\n// defines a DB schema that includes Orders and Books tables\n// and a secondary index: 'BookIdIndex' for the Orders table\npublic class Schema : IDbSchema\n{\n    public BooksSchema Books { get; }\n    public OrdersSchema Orders { get; }\n    \n    public Schema()\n    {\n        Books = new BooksSchema\n        {\n            Table = StereoDb.CreateTable\u003cint, Book\u003e(\"books\")\n        };\n\n        var ordersTable = StereoDb.CreateTable\u003cGuid, Order\u003e(\"orders\");\n\n        Orders = new OrdersSchema()\n        {\n            Table = ordersTable,\n            BookIdIndex = ordersTable.AddValueIndex(order =\u003e order.BookId)\n        };\n    }\n    \n    public IEnumerable\u003cITable\u003e AllTables =\u003e [Orders.Table, Books.Table];\n}\n\npublic static class Demo\n{\n    public static void Run()\n    {\n        var db = StereoDb.Create(new Schema(), StereoDbSettings.Default);\n\n        // 1) adds book\n        // WriteTransaction: it's a read-write transaction: we can query and mutate data\n        \n        db.WriteTransaction(ctx =\u003e\n        {\n            var books = ctx.UseTable(ctx.Schema.Books.Table);\n        \n            foreach (var id in Enumerable.Range(0, 10))\n            {\n                var book = new Book {Id = id, Title = $\"book_{id}\", Quantity = 1};\n                books.Set(book);\n            }\n        });\n               \n        // 2) creates an order\n        // WriteTransaction: it's a read-write transaction: we can query and mutate data\n        \n        db.WriteTransaction(ctx =\u003e\n        {\n            var books = ctx.UseTable(ctx.Schema.Books.Table);\n            var orders = ctx.UseTable(ctx.Schema.Orders.Table);\n        \n            foreach (var id in books.GetIds())\n            {\n                if (books.TryGet(id, out var book) \u0026\u0026 book.Quantity \u003e 0)\n                {\n                    var order = new Order {Id = Guid.NewGuid(), BookId = id, Quantity = 1};\n                    var updatedBook = book with { Quantity = book.Quantity - 1 };\n                    \n                    books.Set(updatedBook);\n                    orders.Set(order);\n                }\n            }\n        });\n                \n        // 3) query book and orders\n        // ReadTransaction: it's a read-only transaction: we can query multiple tables at once\n        \n        var result = db.ReadTransaction(ctx =\u003e\n        {\n            var books = ctx.UseTable(ctx.Schema.Books.Table);\n            var bookIdIndex = ctx.Schema.Orders.BookIdIndex;\n        \n            if (books.TryGet(1, out var book))\n            {\n                var orders = bookIdIndex.Find(book.Id).ToArray();\n                return (book, orders);\n            }\n            \n            return (null, null);\n        });    \n    }    \n}\n```\n\n#### F# API\nF# API has some benefits over C# API, mainly in expressiveness and type safety: \n\n- [Anonymous Records](https://learn.microsoft.com/en-us/dotnet/fsharp/language-reference/anonymous-records) - It provides in place schema definition. You don't need to define extra types for schema as you do with C#. Also, it helps you model efficient **(zero-cost, since it supports [structs](https://learn.microsoft.com/en-us/dotnet/fsharp/language-reference/structs))** and expressive - return result type. \n- [ValueOption\u003c'T\u003e](https://learn.microsoft.com/en-us/dotnet/fsharp/language-reference/value-options) - It's used for StereoDB API to model emptiness in a type safe manner. Also, it's a **zero-cost abstraction** since it's struct.\n- [Computation Expression](https://learn.microsoft.com/en-us/dotnet/fsharp/language-reference/computation-expressions) - It helps to express multiple if \u0026 else checks on emptiness/null for ValueOption\u003c'T\u003e, into a single **voption { }** expression. To use **voption { }**, [FsToolkit.ErrorHandling](https://github.com/demystifyfp/FsToolkit.ErrorHandling) should be installed. In the case of **voption {}**, it's also a **zero-cost abstraction**, the compiler generates optimized code without allocations.\n\n```fsharp\nopen System\nopen FsToolkit.ErrorHandling\nopen StereoDB\nopen StereoDB.FSharp\n\ntype Book = {\n    Id: int\n    Title: string\n    Quantity: int    \n}\n\ntype Order = {\n    Id: Guid\n    BookId: int\n    Quantity: int    \n}\n\n// defines a DB schema that includes Orders and Books tables\n// and a secondary index: 'BookIdIndex' for the Orders table\ntype Schema() =\n    let _books = {| Table = StereoDb.createTable\u003cint, Book\u003e(\"books\") |}\n    \n    let _ordersTable = StereoDb.createTable\u003cGuid, Order\u003e(\"orders\")\n    let _orders = {|\n        Table = _ordersTable\n        BookIdIndex = _ordersTable.AddValueIndex(fun order -\u003e order.BookId)\n    |}\n    \n    member this.Books = _books\n    member this.Orders = _orders\n    \n    interface IDbSchema with\n        member this.AllTables = [_books.Table; _orders.Table] \n\nlet test () =\n    let db = StereoDb.create(Schema(), StereoDbSettings.Default)\n\n    // 1) adds book\n    // WriteTransaction: it's a read-write transaction: we can query and mutate data\n\n    db.WriteTransaction(fun ctx -\u003e\n        let books = ctx.UseTable(ctx.Schema.Books.Table)\n\n        let bookId = 1\n        let book = { Id = bookId; Title = \"book_1\"; Quantity = 1 }\n        books.Set book\n    )\n\n    // 2) creates an order\n    // WriteTransaction: it's a read-write transaction: we can query and mutate data\n\n    db.WriteTransaction(fun ctx -\u003e\n        let books = ctx.UseTable(ctx.Schema.Books.Table)\n        let orders = ctx.UseTable(ctx.Schema.Orders.Table)        \n        \n        voption {\n            let bookId = 1\n            let! book = books.Get bookId\n\n            if book.Quantity \u003e 0 then\n                let order = { Id = Guid.NewGuid(); BookId = bookId; Quantity = 1 }\n                let updatedBook = { book with Quantity = book.Quantity - 1 }\n                \n                books.Set updatedBook\n                orders.Set order                    \n        }\n        |\u003e ignore                     \n    )\n\n    // 3) query book and orders\n    // ReadTransaction: it's a read-only transaction: we can query multiple tables at once\n\n    let result = db.ReadTransaction(fun ctx -\u003e\n        let books = ctx.UseTable(ctx.Schema.Books.Table)\n        let bookIdIndex = ctx.Schema.Orders.BookIdIndex\n        \n        voption {\n            let bookId = 1\n            let! book = books.Get 1\n            let orders = book.Id |\u003e bookIdIndex.Find |\u003e Seq.toArray\n            \n            return struct {| Book = book; Orders = orders |}\n        }\n    )    \n```\n\n#### Transactions\nStereoDB transactions allow the execution of a group of commands in a single step. StereoDB provides Read-Only and Read-Write transactions.\n- Read-Only allows you only read data. **Also, they are multithreaded.**\n- Read-Write allows you read and write data. **They are running in a single-thread fashion.**\n\nWhat to expect from transactions in StereoDB:\n- they are blazingly fast and cheap to execute.\n- they guarantee you atomic and consistent updates (you can update several tables including secondary indexes in one transaction and no other concurrent transaction will read your data partially; the transaction cannot be observed to be in progress by another database client).\n- they don't support rollback since supporting rollbacks would have a significant impact on the simplicity and performance of StereoDB.\n\nIn terms of ACID, StereoDB provides:\nTBD\n\n##### How to deal without rollbacks?\n- we suggest to use only immutable data types to model your data. In C#/F# you can use records/structs to achieve this. Immutable data types allow you to ignore partial failures while updating any data record in memory.\n- run all necessary validation before updating data in tables. Mutating your database should be the latest transaction step after all necessary validations are passed.\n\n```csharp\n// it's an example of WriteTransaction\ndb.WriteTransaction(ctx =\u003e\n{\n    var books = ctx.UseTable(ctx.Schema.Books.Table);    \n\n    // read record\n    var bookId = 42;\n    if (books.TryGet(bookId, out var book) \u0026\u0026 book.Quantity \u003e 0)\n    {\n        // update record (it's immutable type, so the book instance wasn't mutated)        \n        var updatedBook = book with { Quantity = book.Quantity - 1 };\n\n        // and only after this you can safely mutate you state in database\n        books.Set(updatedBook);\n    }    \n});\n```\n\n#### Secondary indexes\nTBD\n\n#### Best practices\nTBD\n\u003c!-- - closures (allocation)\n- use immutable types\n- secondary index IEnumerable convert to array for response\n- GC settings\n- Serializations, HTTP 2 --\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstereodb%2Fstereodb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstereodb%2Fstereodb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstereodb%2Fstereodb/lists"}