Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/stereodb/stereodb
Ultrafast and lightweight in-process memory database written in F# that supports: transactions, secondary indexes, persistence, and data size larger than RAM.
https://github.com/stereodb/stereodb
caching csharp database dotnet dotnet-core fsharp hacktoberfest in-memory-database
Last synced: about 6 hours ago
JSON representation
Ultrafast and lightweight in-process memory database written in F# that supports: transactions, secondary indexes, persistence, and data size larger than RAM.
- Host: GitHub
- URL: https://github.com/stereodb/stereodb
- Owner: StereoDB
- License: apache-2.0
- Created: 2023-01-18T15:19:57.000Z (almost 2 years ago)
- Default Branch: dev
- Last Pushed: 2024-08-28T13:43:59.000Z (3 months ago)
- Last Synced: 2024-10-20T01:58:19.927Z (24 days ago)
- Topics: caching, csharp, database, dotnet, dotnet-core, fsharp, hacktoberfest, in-memory-database
- Language: F#
- Homepage:
- Size: 952 KB
- Stars: 192
- Watchers: 7
- Forks: 4
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://github.com/StereoDB/StereoDB/actions/workflows/build.yml)
[](https://www.nuget.org/packages/stereodb/)
#### StereoDB
Ultrafast and lightweight in-process memory database written in F# that supports: transactions, secondary indexes, persistence, and data size larger than RAM. The primary use case for this database is building Stateful Services (API or ETL Worker) that keep all data in memory and can provide millions of RPS from a single node.
Supported features:
- [x] C# and F# API
- [x] Basic SQL support
- [x] Transactions (read-only, read-write)
- [x] Secondary Indexes
- [x] Value Index (hash-based index)
- [x] Range Scan Index
- [ ] Data size larger than RAM
- [ ] Data persistence
- [ ] Distributed mode
- [ ] Server and client discovery
- [ ] Range-based sharding
#### Intro to Stateful Services
#### Benchmarks
Pure KV workload benchmark (in-process only, without persistence). In [this benchmark](https://github.com/StereoDB/StereoDB/blob/dev/benchmarks/StereoDB.Benchmarks/Benchmarks/StereoDbBenchmark.cs), **we run concurrently 3 million random reads and 100K random writes in 892 ms**.
```
BenchmarkDotNet=v0.13.5, OS=Windows 11 (10.0.22621.2134/22H2/2022Update/SunValley2)
AMD Ryzen 7 5800H with Radeon Graphics, 1 CPU, 16 logical and 8 physical cores
.NET SDK=7.0.400
[Host] : .NET 7.0.10 (7.0.1023.36312), X64 RyuJIT AVX2
DefaultJob : .NET 7.0.10 (7.0.1023.36312), X64 RyuJIT AVX2
| Method | ReadThreadCount | WriteThreadCount | UsersCount | DbReadCount | DbWriteCount | Mean | Error | StdDev | Allocated |
|---------- |---------------- |----------------- |----------- |------------ |------------- |---------:|---------:|---------:|----------:|
| ReadWrite | 30 | 30 | 4000000 | 3000000 | 100000 | 891.9 ms | 17.75 ms | 35.86 ms | 13.12 KB |
```
#### C# API
```csharp
using System;
using StereoDB;
using StereoDB.CSharp;
public record Book
{
public int Id { get; init; }
public string Title { get; init; }
public int Quantity { get; init; }
}
public record Order
{
public Guid Id { get; init; }
public int BookId { get; init; }
public int Quantity { get; init; }
}
public class BooksSchema
{
public ITable Table { get; init; }
}
public class OrdersSchema
{
public ITable Table { get; init; }
public IValueIndex BookIdIndex { get; init; }
}
// defines a DB schema that includes Orders and Books tables
// and a secondary index: 'BookIdIndex' for the Orders table
public class Schema : IDbSchema
{
public BooksSchema Books { get; }
public OrdersSchema Orders { get; }
public Schema()
{
Books = new BooksSchema
{
Table = StereoDb.CreateTable("books")
};
var ordersTable = StereoDb.CreateTable("orders");
Orders = new OrdersSchema()
{
Table = ordersTable,
BookIdIndex = ordersTable.AddValueIndex(order => order.BookId)
};
}
public IEnumerable AllTables => [Orders.Table, Books.Table];
}
public static class Demo
{
public static void Run()
{
var db = StereoDb.Create(new Schema(), StereoDbSettings.Default);
// 1) adds book
// WriteTransaction: it's a read-write transaction: we can query and mutate data
db.WriteTransaction(ctx =>
{
var books = ctx.UseTable(ctx.Schema.Books.Table);
foreach (var id in Enumerable.Range(0, 10))
{
var book = new Book {Id = id, Title = $"book_{id}", Quantity = 1};
books.Set(book);
}
});
// 2) creates an order
// WriteTransaction: it's a read-write transaction: we can query and mutate data
db.WriteTransaction(ctx =>
{
var books = ctx.UseTable(ctx.Schema.Books.Table);
var orders = ctx.UseTable(ctx.Schema.Orders.Table);
foreach (var id in books.GetIds())
{
if (books.TryGet(id, out var book) && book.Quantity > 0)
{
var order = new Order {Id = Guid.NewGuid(), BookId = id, Quantity = 1};
var updatedBook = book with { Quantity = book.Quantity - 1 };
books.Set(updatedBook);
orders.Set(order);
}
}
});
// 3) query book and orders
// ReadTransaction: it's a read-only transaction: we can query multiple tables at once
var result = db.ReadTransaction(ctx =>
{
var books = ctx.UseTable(ctx.Schema.Books.Table);
var bookIdIndex = ctx.Schema.Orders.BookIdIndex;
if (books.TryGet(1, out var book))
{
var orders = bookIdIndex.Find(book.Id).ToArray();
return (book, orders);
}
return (null, null);
});
}
}
```
#### F# API
F# API has some benefits over C# API, mainly in expressiveness and type safety:
- [Anonymous Records](https://learn.microsoft.com/en-us/dotnet/fsharp/language-reference/anonymous-records) - It provides in place schema definition. You don't need to define extra types for schema as you do with C#. Also, it helps you model efficient **(zero-cost, since it supports [structs](https://learn.microsoft.com/en-us/dotnet/fsharp/language-reference/structs))** and expressive - return result type.
- [ValueOption<'T>](https://learn.microsoft.com/en-us/dotnet/fsharp/language-reference/value-options) - It's used for StereoDB API to model emptiness in a type safe manner. Also, it's a **zero-cost abstraction** since it's struct.
- [Computation Expression](https://learn.microsoft.com/en-us/dotnet/fsharp/language-reference/computation-expressions) - It helps to express multiple if & else checks on emptiness/null for ValueOption<'T>, into a single **voption { }** expression. To use **voption { }**, [FsToolkit.ErrorHandling](https://github.com/demystifyfp/FsToolkit.ErrorHandling) should be installed. In the case of **voption {}**, it's also a **zero-cost abstraction**, the compiler generates optimized code without allocations.
```fsharp
open System
open FsToolkit.ErrorHandling
open StereoDB
open StereoDB.FSharp
type Book = {
Id: int
Title: string
Quantity: int
}
type Order = {
Id: Guid
BookId: int
Quantity: int
}
// defines a DB schema that includes Orders and Books tables
// and a secondary index: 'BookIdIndex' for the Orders table
type Schema() =
let _books = {| Table = StereoDb.createTable("books") |}
let _ordersTable = StereoDb.createTable("orders")
let _orders = {|
Table = _ordersTable
BookIdIndex = _ordersTable.AddValueIndex(fun order -> order.BookId)
|}
member this.Books = _books
member this.Orders = _orders
interface IDbSchema with
member this.AllTables = [_books.Table; _orders.Table]
let test () =
let db = StereoDb.create(Schema(), StereoDbSettings.Default)
// 1) adds book
// WriteTransaction: it's a read-write transaction: we can query and mutate data
db.WriteTransaction(fun ctx ->
let books = ctx.UseTable(ctx.Schema.Books.Table)
let bookId = 1
let book = { Id = bookId; Title = "book_1"; Quantity = 1 }
books.Set book
)
// 2) creates an order
// WriteTransaction: it's a read-write transaction: we can query and mutate data
db.WriteTransaction(fun ctx ->
let books = ctx.UseTable(ctx.Schema.Books.Table)
let orders = ctx.UseTable(ctx.Schema.Orders.Table)
voption {
let bookId = 1
let! book = books.Get bookId
if book.Quantity > 0 then
let order = { Id = Guid.NewGuid(); BookId = bookId; Quantity = 1 }
let updatedBook = { book with Quantity = book.Quantity - 1 }
books.Set updatedBook
orders.Set order
}
|> ignore
)
// 3) query book and orders
// ReadTransaction: it's a read-only transaction: we can query multiple tables at once
let result = db.ReadTransaction(fun ctx ->
let books = ctx.UseTable(ctx.Schema.Books.Table)
let bookIdIndex = ctx.Schema.Orders.BookIdIndex
voption {
let bookId = 1
let! book = books.Get 1
let orders = book.Id |> bookIdIndex.Find |> Seq.toArray
return struct {| Book = book; Orders = orders |}
}
)
```
#### Transactions
StereoDB transactions allow the execution of a group of commands in a single step. StereoDB provides Read-Only and Read-Write transactions.
- Read-Only allows you only read data. **Also, they are multithreaded.**
- Read-Write allows you read and write data. **They are running in a single-thread fashion.**
What to expect from transactions in StereoDB:
- they are blazingly fast and cheap to execute.
- they guarantee you atomic and consistent updates (you can update several tables including secondary indexes in one transaction and no other concurrent transaction will read your data partially; the transaction cannot be observed to be in progress by another database client).
- they don't support rollback since supporting rollbacks would have a significant impact on the simplicity and performance of StereoDB.
In terms of ACID, StereoDB provides:
TBD
##### How to deal without rollbacks?
- we suggest to use only immutable data types to model your data. In C#/F# you can use records/structs to achieve this. Immutable data types allow you to ignore partial failures while updating any data record in memory.
- run all necessary validation before updating data in tables. Mutating your database should be the latest transaction step after all necessary validations are passed.
```csharp
// it's an example of WriteTransaction
db.WriteTransaction(ctx =>
{
var books = ctx.UseTable(ctx.Schema.Books.Table);
// read record
var bookId = 42;
if (books.TryGet(bookId, out var book) && book.Quantity > 0)
{
// update record (it's immutable type, so the book instance wasn't mutated)
var updatedBook = book with { Quantity = book.Quantity - 1 };
// and only after this you can safely mutate you state in database
books.Set(updatedBook);
}
});
```
#### Secondary indexes
TBD
#### Best practices
TBD