https://github.com/trueai-org/ultra-kv
Single File Lightning-Fast Key-Value Storage Engine
https://github.com/trueai-org/ultra-kv
keyvaluestore litedb single-file sqlite storage
Last synced: 3 months ago
JSON representation
Single File Lightning-Fast Key-Value Storage Engine
- Host: GitHub
- URL: https://github.com/trueai-org/ultra-kv
- Owner: trueai-org
- License: apache-2.0
- Created: 2025-06-12T04:07:36.000Z (4 months ago)
- Default Branch: main
- Last Pushed: 2025-06-12T04:45:46.000Z (4 months ago)
- Last Synced: 2025-06-12T05:34:45.969Z (4 months ago)
- Topics: keyvaluestore, litedb, single-file, sqlite, storage
- Language: C#
- Homepage:
- Size: 136 KB
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# π UltraKV - Lightning-Fast Key-Value Storage Engine
[](https://choosealicense.com/licenses/mit/)
[](https://dotnet.microsoft.com/download/dotnet/8.0)
[](https://docs.microsoft.com/en-us/dotnet/csharp/)
[](#performance-benchmarks)**An ultra-high performance, enterprise-grade key-value storage solution designed for the .NET ecosystem**
[English](README.md) | [δΈζζζ‘£](README_CN.md)
## π― Minimal Code, Maximum Performance
### **π Less than 1,000 Lines of Code for a Complete Database Engine**
*Proving that simplicity and performance can coexist*
One of the most remarkable aspects of **UltraKV** is achieving enterprise-grade database performance with an incredibly compact codebase. The entire **UltraKV** engine core is implemented in **less than 1,000 lines of C# code**, demonstrating the power of focused, efficient design. Database initializing only requires **64 bytes**.
### π‘ Philosophy: "Perfect is the Enemy of Good"
UltraKV proves that:
- **Less code = Less bugs**
- **Simple design = Better performance**
- **Focused scope = Reliable execution**
- **Clear logic = Easy maintenance**> *"The best code is no code at all. The second best is code so simple and efficient that it feels like no code."*
## Installation
[More Document / WIKI](https://github.com/trueai-org/ultra-kv/wiki)
```bash
dotnet add package UltraKVusing var engine = new UltraKVEngine("test.db");
engine.Set("key1", "value1");
var value = engine.Get("key1");
engine.Remove("key1");
engine.Flush();
```## π Table of Contents
- [π Project Overview](#-project-overview)
- [π₯ Key Features](#-key-features)
- [β‘ Performance Benchmarks](#-performance-benchmarks)
- [ποΈ Architecture Design](#οΈ-architecture-design)
- [π Quick Start](#-quick-start)
- [π Detailed Usage Guide](#-detailed-usage-guide)
- [π§ Advanced Configuration](#-advanced-configuration)
- [π Performance Testing](#-performance-testing)
- [π Security Features](#-security-features)
- [π οΈ Best Practices](#οΈ-best-practices)
- [π€ Contributing](#-contributing)
- [π License](#-license)## π Project Overview
**UltraKV** is a modern, single file, high-performance key-value storage system.
### π― Design Goals
- **π Extreme Performance**: Supporting millions of ops/sec for read/write operations per instance
- **π Data Security**: Enterprise-grade encryption, compression, and data integrity protection
- **π‘οΈ High Reliability**: Atomic transactions, data persistence, and fault recovery
- **βοΈ Highly Configurable**: Flexible configuration options for different application scenarios
- **π Scalable**: Multi-engine management and horizontal scaling support## π₯ Key Features
### π― UltraKV Engine Features
| Feature Category | Specific Functionality | Description |
|-----------------|------------------------|-------------|
| **π Performance Optimization** | In-Memory Index + Disk Storage | Dual-guarantee high-performance architecture |
| | Batch Operations Support | Efficient batch read/write and delete operations |
| | Smart Buffering Mechanism | Configurable write buffer |
| | Concurrency Control | Thread-safe concurrent access |
| **πΎ Storage Management** | Automatic Space Reclaim | Intelligent disk space compaction |
| | Multiple Update Modes | Append mode and replace mode |
| | Scheduled Persistence | Configurable auto-flush strategy |
| | Optimized File Format | Compact binary storage format |
| **π Data Security** | Multiple Encryption Algorithms | AES256-GCM, ChaCha20-Poly1305 |
| | Compression Support | LZ4, Zstd, Snappy, Gzip, etc. |
| | Data Integrity Verification | Multiple hash algorithms for validation |
| | Atomicity Guarantee | Write locks ensure transaction atomicity |### ποΈ File Storage Structure
#### UltraKV Storage Format
```
ββββββββββββββββββββββββββββββββββββββββββββββββββββ
β 1. Database Header (Fixed 64 bytes) β β File Start β
ββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β 2. Value Data Area β
β ββ Data Record 1 (Variable length) β
β ββ Data Record 2 (Variable length) β
β ββ ... β
ββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β 3. Index Data Area [Start, End] β β File End β
β ββ Index Entry 1 (Key + Position info) β
β ββ Index Entry 2 (Key + Position info) β
β ββ ... β
ββββββββββββββββββββββββββββββββββββββββββββββββββββ
```## β‘ Performance Benchmarks
### π UltraKV Performance Benchmark
> Test Environment: .NET 8.0, Windows 11, SSD Storage
| Operation Type | Performance (ops/sec) | Notes |
|---------------|----------------------|-------|
| **Sequential Write** | **462,963** | Optimized for high-frequency writes |
| **Batch Insert** | **564,972** | Enhanced batch operation performance |
| **Random Read** | **632,911** | Accelerated by in-memory index |
| **Contains Check** | **25,000,000** | Ultra-fast memory operations |
| **Data Delete** | **833,333** | Efficient delete operations |
| **Batch Delete** | **1,562,500** | Outstanding batch delete performance |
| **Data Update** | **333,333** | In-place update optimization |
| **Random Access** | **500,000** | Excellent random access performance |#### π§ In-Memory Mode Performance
> Test Environment: .NET 8.0, Windows 11, Average of 10 rounds, Memory mode operations
| Operation Type | Performance (ops/sec) | Notes |
|---------------|----------------------|-------|
| **Write Performance** | **1,958,190** | Ultra-high speed memory writes |
| **Read Performance** | **6,100,949** | Lightning-fast memory reads |
| **ContainsKey Performance** | **7,844,485** | Optimized key existence checks |
| **Delete Performance** | **5,984,823** | Efficient memory deletions |> *Memory mode provides 3-8x performance improvement over file-based storage, ideal for cache and temporary data scenarios, Automatic timed persistence, and the memory mode can be turned off at any time*
## π Quick Start
### π¦ Installation
```xml
```
### π§ Basic Usage
#### UltraKV Basic Example
```csharp
using UltraKV;// Create engine manager
using var manager = new UltraKVManager("./data");// Get engine instance
var engine = manager.GetEngine("my_database");// Basic operations
engine.Set("user:1001", "John Doe");
engine.Set("user:1002", "Jane Smith");// Read data
var user = engine.Get("user:1001"); // Returns: "John Doe"
var exists = engine.ContainsKey("user:1001"); // Returns: true// Delete data
engine.Remove("user:1002");// Batch operations
var batch = new Dictionary
{
["product:1"] = "Laptop",
["product:2"] = "Mouse",
["product:3"] = "Keyboard"
};
engine.SetBatch(batch);// Persist data
engine.Flush();
```#### Advanced Configuration Example
```csharp
// Create high-performance configuration
var config = new UltraKVConfig
{
// Compression configuration
CompressionType = CompressionType.LZ4,
// Encryption configuration
EncryptionType = EncryptionType.AES256GCM,
EncryptionKey = "MySecureKey32BytesLong!@#$%^&*()",
// Performance configuration
FileStreamBufferSizeKB = 1024, // 1MB buffer
WriteBufferSizeKB = 2048, // 2MB write buffer
FlushInterval = 10, // 10-second auto-flush
// Maintenance configuration
AutoCompactEnabled = true, // Enable auto-compaction
AutoCompactThreshold = 30, // 30% fragmentation triggers compaction
// File update mode
FileUpdateMode = FileUpdateMode.Append // Append mode for higher performance
};var engine = manager.GetEngine("high_performance_db", config);
```## π Detailed Usage Guide
### π§ Configuration Options Explained
#### UltraKVConfig Core Configuration
```csharp
public class UltraKVConfig
{
// π― Performance Related
public bool EnableMemoryMode { get; set; } = false; // Memory mode
public int FileStreamBufferSizeKB { get; set; } = 64; // File buffer
public bool EnableWriteBuffer { get; set; } = true; // Write buffer
public int WriteBufferSizeKB { get; set; } = 1024; // Buffer size
// π Security Related
public CompressionType CompressionType { get; set; } // Compression algorithm
public EncryptionType EncryptionType { get; set; } // Encryption algorithm
public HashType HashType { get; set; } = HashType.XXH3; // Hash algorithm
public string? EncryptionKey { get; set; } // Encryption key
// π Maintenance Related
public bool AutoCompactEnabled { get; set; } = false; // Auto-compaction
public byte AutoCompactThreshold { get; set; } = 50; // Compaction threshold
public ushort FlushInterval { get; set; } = 5; // Flush interval
public FileUpdateMode FileUpdateMode { get; set; } // Update mode
// π‘οΈ Validation Related
public bool EnableUpdateValidation { get; set; } = false; // Update validation
public int MaxKeyLength { get; set; } = 4096; // Maximum key length
}
```### π Lifecycle Management
```csharp
// Engine manager supports multiple engines
using var manager = new UltraKVManager("./databases");// Create engines for different purposes
var userEngine = manager.GetEngine("users", UltraKVConfig.Default);
var sessionEngine = manager.GetEngine("sessions", UltraKVConfig.Minimal);
var cacheEngine = manager.GetEngine("cache", new UltraKVConfig
{
FlushInterval = 30, // Cache data can flush less frequently
EnableMemoryMode = true // Enable memory mode for cache
});// Batch flush
manager.FlushAll();// Close specific engine
manager.CloseEngine("cache");// Get engine list
var engineNames = manager.GetEngineNames();
```### π Performance Monitoring
```csharp
// Get engine statistics
var stats = engine.GetStats();
Console.WriteLine($"Record Count: {stats.RecordCount}");
Console.WriteLine($"Deleted Count: {stats.DeletedCount}");
Console.WriteLine($"File Size: {stats.FileSize / 1024.0 / 1024.0:F2} MB");
Console.WriteLine($"Deletion Ratio: {stats.DeletionRatio:P1}");
Console.WriteLine($"Compaction Recommended: {stats.ShrinkRecommended}");// Manually trigger compaction
if (engine.ShouldShrink())
{
engine.Compact(fullRebuild: false);
}
```## π§ Advanced Configuration
### π Performance Optimization Configuration
#### High Throughput Write Scenarios
```csharp
var highThroughputConfig = new UltraKVConfig
{
FileUpdateMode = FileUpdateMode.Append, // Append mode for best performance
WriteBufferSizeKB = 4096, // 4MB large buffer
FileStreamBufferSizeKB = 2048, // 2MB file buffer
FlushInterval = 30, // Longer flush interval
AutoCompactThreshold = 70 // Higher compaction threshold
};
```#### Low Latency Read Scenarios
```csharp
var lowLatencyConfig = new UltraKVConfig
{
EnableMemoryMode = true, // Memory mode for lowest latency
FlushInterval = 5, // Frequent flushing for data safety
EnableUpdateValidation = true // Enable validation for data correctness
};
```#### Storage Space Sensitive Scenarios
```csharp
var compactConfig = new UltraKVConfig
{
CompressionType = CompressionType.Zstd, // Best compression ratio
AutoCompactEnabled = true, // Enable auto-compaction
AutoCompactThreshold = 20, // Low threshold triggers compaction
FileUpdateMode = FileUpdateMode.Replace // Replace mode reduces fragmentation
};
```### π Security Configuration Examples
#### Enterprise-Grade Security Configuration
```csharp
var secureConfig = UltraKVConfig.Secure("MyEnterprise256BitSecretKey!@#");
// Equivalent to:
var secureConfig = new UltraKVConfig
{
CompressionType = CompressionType.Gzip,
EncryptionType = EncryptionType.AES256GCM,
EncryptionKey = "MyEnterprise256BitSecretKey!@#",
HashType = HashType.SHA256,
EnableUpdateValidation = true
};
```#### Debug and Development Configuration
```csharp
var debugConfig = UltraKVConfig.Debug; // Enable all validation options
```### π Compression Algorithm Selection Guide
| Algorithm | Compression Ratio | Speed | Use Case |
|-----------|------------------|-------|----------|
| **LZ4** | Medium | Extremely Fast | High-performance requirements |
| **Zstd** | Excellent | Fast | Balance performance and compression |
| **Snappy** | Medium | Extremely Fast | Google ecosystem |
| **Gzip** | Good | Medium | General-purpose compression |
| **Brotli** | Excellent | Slower | Web application optimization |### π Encryption Algorithm Selection Guide
| Algorithm | Security Level | Performance | Use Case |
|-----------|---------------|-------------|----------|
| **AES256-GCM** | Extremely High | Excellent | Enterprise applications |
| **ChaCha20-Poly1305** | Extremely High | Excellent | Mobile device optimization |## π Performance Testing
### π§ͺ Built-in Performance Tests
The project includes a complete performance testing suite. You can run the following tests:
```bash
# Clone the project
git clone https://github.com/trueai-org/UltraKV.git
cd UltraKV# Run UltraKV performance tests
dotnet run --project src/UltraKV --configuration Release# Run comparison tests
dotnet test src/UltraKV.Tests --configuration Release
```### π Custom Performance Testing
```csharp
// Performance testing example
public async Task BenchmarkWritePerformance()
{
using var manager = new UltraKVManager("./benchmark");
var engine = manager.GetEngine("test");
const int iterations = 100_000;
var stopwatch = Stopwatch.StartNew();
for (int i = 0; i < iterations; i++)
{
engine.Set($"key_{i}", $"value_{i}");
if (i % 1000 == 0) // Flush every 1000 operations
{
engine.Flush();
}
}
engine.Flush();
stopwatch.Stop();
var opsPerSecond = iterations * 1000.0 / stopwatch.ElapsedMilliseconds;
Console.WriteLine($"Write Performance: {opsPerSecond:N0} ops/sec");
}
```### π Compaction Performance Testing
```csharp
// Test compaction operation performance
public void BenchmarkCompactPerformance()
{
using var engine = new UltraKVEngine("./compact_test.db");
// Write large amount of data
for (int i = 0; i < 50_000; i++)
{
engine.Set($"key_{i}", new string('x', 1024)); // 1KB data
}
// Delete 50% of data to create fragmentation
for (int i = 0; i < 50_000; i += 2)
{
engine.Remove($"key_{i}");
}
var beforeSize = new FileInfo("./compact_test.db").Length;
var beforeStats = engine.GetStats();
var stopwatch = Stopwatch.StartNew();
engine.Compact(fullRebuild: false);
stopwatch.Stop();
var afterSize = new FileInfo("./compact_test.db").Length;
var afterStats = engine.GetStats();
Console.WriteLine($"Compaction Time: {stopwatch.ElapsedMilliseconds}ms");
Console.WriteLine($"File Size: {beforeSize / 1024 / 1024}MB -> {afterSize / 1024 / 1024}MB");
Console.WriteLine($"Space Saved: {(1 - (double)afterSize / beforeSize):P1}");
}
```## π Security Features
### π Data Encryption
UltraKV supports industry-standard encryption algorithms:
- **AES256-GCM**: Widely recognized enterprise-grade encryption standard
- **ChaCha20-Poly1305**: Modern high-performance encryption algorithm```csharp
// Enable encrypted storage
var encryptedEngine = manager.GetEngine("secure_data", new UltraKVConfig
{
EncryptionType = EncryptionType.AES256GCM,
EncryptionKey = "MySecure32ByteEncryptionKey12345"
});// Data will be automatically encrypted when stored
encryptedEngine.Set("sensitive_data", "confidential_information");
```### π Data Integrity
Multiple data integrity protection mechanisms:
```csharp
var validatedConfig = new UltraKVConfig
{
EnableUpdateValidation = true, // Enable write validation
HashType = HashType.SHA256, // Use SHA256 for data verification
};
```Supported hash algorithms:
- **XXH3**: Ultra-fast hash, default choice
- **SHA256**: Cryptographic-grade hash
- **BLAKE3**: Modern high-performance hash
- **XXH128**: 128-bit hash with low collision rate## π οΈ Best Practices
### π‘ Performance Optimization Recommendations
1. **Buffer Configuration**
```csharp
// Adjust buffer size based on memory availability
var config = new UltraKVConfig
{
FileStreamBufferSizeKB = Environment.Is64BitProcess ? 1024 : 256,
WriteBufferSizeKB = Environment.Is64BitProcess ? 4096 : 1024
};
```2. **Batch Operation Optimization**
```csharp
// Use batch operations to improve performance
var batch = new Dictionary();
for (int i = 0; i < 10000; i++)
{
batch[$"key_{i}"] = $"value_{i}";
}
engine.SetBatch(batch); // Much faster than individual Set operations
```3. **Reasonable Flush Strategy**
```csharp
// High-frequency write scenarios
var highWriteConfig = new UltraKVConfig
{
FlushInterval = 30, // Flush every 30 seconds
WriteBufferSizeKB = 8192 // 8MB buffer
};// Low-latency scenarios
var lowLatencyConfig = new UltraKVConfig
{
FlushInterval = 1, // Flush every 1 second
EnableUpdateValidation = true // Enable validation
};
```### π§ Maintenance and Monitoring
1. **Regular Statistics Monitoring**
```csharp
// Regularly check engine status
var timer = new Timer(async _ =>
{
var stats = engine.GetStats();
if (stats.DeletionRatio > 0.3) // Deletion ratio exceeds 30%
{
Console.WriteLine("Compaction operation recommended");
if (engine.ShouldShrink())
{
engine.Compact();
}
}
}, null, TimeSpan.Zero, TimeSpan.FromMinutes(5));
```2. **Graceful Shutdown Handling**
```csharp
// Ensure data safety when application shuts down
AppDomain.CurrentDomain.ProcessExit += (sender, e) =>
{
manager.FlushAll(); // Flush all engines
manager.Dispose(); // Release resources
};
```### π¨ Error Handling
```csharp
try
{
engine.Set("key", "value");
}
catch (InvalidOperationException ex) when (ex.Message.Contains("disposed"))
{
// Engine has been disposed
Console.WriteLine("Engine is closed, please reinitialize");
}
catch (ArgumentException ex) when (ex.Message.Contains("EncryptionKey"))
{
// Encryption configuration error
Console.WriteLine("Encryption key configuration error");
}
catch (IOException ex)
{
// Disk IO error
Console.WriteLine($"Disk operation failed: {ex.Message}");
}
```## π Data Migration and Backup
### π€ Data Export
```csharp
// Export all data
public void ExportData(UltraKVEngine engine, string backupFile)
{
var allKeys = engine.GetAllKeys();
var backup = new Dictionary();
foreach (var key in allKeys)
{
var value = engine.Get(key);
if (value != null)
{
backup[key] = value;
}
}
var json = JsonSerializer.Serialize(backup, new JsonSerializerOptions
{
WriteIndented = true
});
File.WriteAllText(backupFile, json);
}
```### π₯ Data Import
```csharp
// Restore data from backup
public void ImportData(UltraKVEngine engine, string backupFile)
{
var json = File.ReadAllText(backupFile);
var backup = JsonSerializer.Deserialize>(json);
if (backup != null)
{
engine.SetBatch(backup);
engine.Flush();
}
}
```## π Integration with Other Technologies
### π ASP.NET Core Integration
```csharp
// Startup.cs or Program.cs
services.AddSingleton>(provider =>
new UltraKVManager("./app_data"));services.AddSingleton(provider =>
{
var manager = provider.GetService>();
return new UltraKVCache(manager.GetEngine("cache"));
});
```### π§ Custom Cache Implementation
```csharp
public class UltraKVCache : IMemoryCache
{
private readonly UltraKVEngine _engine;
public UltraKVCache(UltraKVEngine engine)
{
_engine = engine;
}
public bool TryGetValue(object key, out object value)
{
value = _engine.Get(key.ToString()!);
return value != null;
}
public ICacheEntry CreateEntry(object key)
{
return new UltraKVCacheEntry(key.ToString()!, _engine);
}
// ... other interface implementations
}
```## π€ Contributing
We welcome all forms of contribution! Please read [CONTRIBUTING.md](CONTRIBUTING.md) for detailed information.
### π Bug Reports
Before submitting an issue, please ensure:
1. Search existing issues
2. Provide detailed error information and reproduction steps
3. Include environment information (.NET version, operating system, etc.)### π‘ Feature Requests
We welcome feature suggestions! Please describe in detail in the issue:
1. Use case for the feature
2. Expected behavior
3. Possible implementation approaches### π§ Development Environment Setup
```bash
# 1. Fork and clone the project
https://github.com/trueai-org/ultra-kv.git
cd ultra-kv# 2. Install dependencies
dotnet restore# 3. Run tests
dotnet test# 4. Build project
dotnet build --configuration Release
```## π References
This project references the following excellent open-source projects:
- [Lightning.NET](https://github.com/CoreyKaylor/Lightning.NET) - LMDB .NET bindings
- [Fast-Persistent-Dictionary](https://github.com/jgric2/Fast-Persistent-Dictionary) - Persistent dictionary implementation
- [Microsoft FASTER](https://github.com/microsoft/FASTER) - High-performance key-value store
- [DBreeze](https://github.com/hhblaze/DBreeze) - .NET embedded database## π Performance Comparison
| Database | Write (ops/s) | Read (ops/s) | Features |
|----------|--------------|-------------|----------|
| **UltraKV UltraKV** | **462,963** | **632,911** | Pure .NET, zero dependencies |
| FASTER | ~400,000 | ~1,000,000 | Microsoft product, memory optimized |
| LevelDB (C++) | ~100,000 | ~200,000 | Google product, battle-tested |
| SQLite | ~50,000 | ~100,000 | Relational, feature-complete |> *Performance data based on benchmark tests on the same hardware environment. Actual performance varies by environment*
## π Roadmap
### π― Near-term Goals (v1.1)
- [ ] Distributed support and cluster mode
- [ ] More compression algorithm support
- [ ] Performance monitoring and metrics export
- [ ] Database repair tools### π Mid-term Goals (v2.0)
- [ ] Support for complex queries and indexing
- [ ] Plugin-based storage backends
- [ ] Cloud-native support
- [ ] Graphical management interface### π Long-term Goals (v3.0)
- [ ] Machine learning optimized performance tuning
- [ ] Automated operations and fault recovery
- [ ] Cross-platform mobile support## π± Community and Support
- π¬ [Discussions](https://github.com/trueai-org/ultra-kv/discussions) - Technical discussions and Q&A
- π§ [Mailing List](mailto:ultrakv@trueai.org) - Official announcements and updates
- π [Issue Tracker](https://github.com/trueai-org/ultra-kv/issues) - Bug reports and feature requests
- π [Wiki](https://github.com/trueai-org/ultra-kv/wiki) - Detailed documentation and tutorials## π License
This project is licensed under the [MIT License](LICENSE).
---
**β If this project helps you, please give us a Star! β**
[π Home](https://github.com/trueai-org/ultra-kv) β’
[π Documentation](https://github.com/trueai-org/ultra-kv/wiki) β’
[π Report Issues](https://github.com/trueai-org/ultra-kv/issues) β’
[π‘ Feature Requests](https://github.com/trueai-org/ultra-kv/issues/new?template=feature_request.md)Copyright Β© 2024 TrueAI.org. All rights reserved.