Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/mohitgupta07/go-hit
A KV store written in go for high throughput, low latency, high scalability and fault tolerance.
https://github.com/mohitgupta07/go-hit
cassandra fault-tolerance golang high-availability high-performance high-scalability high-throughput nosql persistent-storage postgresql redis single-file-write
Last synced: about 1 month ago
JSON representation
A KV store written in go for high throughput, low latency, high scalability and fault tolerance.
- Host: GitHub
- URL: https://github.com/mohitgupta07/go-hit
- Owner: mohitgupta07
- License: gpl-3.0
- Created: 2024-07-09T23:37:33.000Z (6 months ago)
- Default Branch: main
- Last Pushed: 2024-07-23T17:29:08.000Z (5 months ago)
- Last Synced: 2024-10-19T23:24:19.738Z (2 months ago)
- Topics: cassandra, fault-tolerance, golang, high-availability, high-performance, high-scalability, high-throughput, nosql, persistent-storage, postgresql, redis, single-file-write
- Language: Go
- Homepage:
- Size: 94.7 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Go-Hit
This is a KV-store tool implemented in Go.
Simple objective: To make a tool which is super scalable, fast, can handle tons of data for scalability.
To start the server:
```shell
go run cmd/go-hit-server/main.go
```
Simple objective of this is to create the best of all worlds like solution that can support multiple backend storage. Also, it should be concurrent in nature.
**Why do we need this?**
- Easy to digest, fast to read and fault tolerant to restart.
- Can support 100k writes a second and data reload (on restart) should be atleast as fast as 100k read per sec. Note that we are just talking about a simple benchmark of write a {"key":"value"} no matter how we store in rdbms, nosql, file storage, etc.
Benchmark Report:
| Benchmark Name | Operations | Time per Operation | Memory Allocs per Operation | Notes |
|-------------------------------------|------------|--------------------|-----------------------------|-------------------------------|
| **Postgres** | | | | |
| BenchmarkSaveToDisk | 8886 | 138285 ns/op | 462 B/op | |
| BenchmarkLoad | 9631 | 2622185 ns/op | 2046178 B/op | |
| **Postgres with IO concurrency = 10**| | | | |
| BenchmarkSaveToDisk | 29551 | 37869 ns/op | 493 B/op | |
| BenchmarkLoad | 1450 | 7427697 ns/op | 3173033 B/op | |
| **SFW with IO concurrency = 10** | | | | |
| BenchmarkSaveToDisk | 16446 | 104123 ns/op | 782 B/op | |
| BenchmarkSaveToDisk(without cleanup) | 30753 | 35351 ns/op | 4862 B/op | |
| BenchmarkLoad | 1704 | 9426406 ns/op | 3909961 B/op | |
| **Cassandra with IO concurrency = 500** | | | | |
| BenchmarkSaveToDisk | 103744 | 9803 ns/op | 2899 B/op | |
| BenchmarkLoad | 638 | 2193735 ns/op | 132143 B/op | |
One interesting thing to notice about sfw and cassandra load operation is that when we do load operation without benchmarking tool, it works pretty fast