https://github.com/ysmood/storer
On-disk high-performance object storage
https://github.com/ysmood/storer
Last synced: 8 months ago
JSON representation
On-disk high-performance object storage
- Host: GitHub
- URL: https://github.com/ysmood/storer
- Owner: ysmood
- Created: 2019-06-15T21:03:30.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2021-01-06T08:20:48.000Z (about 5 years ago)
- Last Synced: 2024-06-21T16:59:07.018Z (over 1 year ago)
- Language: Go
- Homepage:
- Size: 115 KB
- Stars: 19
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Storer
[](http://godoc.org/github.com/ysmood/storer)
[](https://codecov.io/gh/ysmood/storer)
On-disk high-performance lightweight object storage for golang. This project is based on my research of
the minimum interface to create an efficient indexable database with a key-value store.
It's a proof of concept for [pkg/kvstore](pkg/kvstore/interface.go).
Sometimes a single point of failure also means low resource wasting and easy to debug.
So it's not a bad idea to use embedded DB at the early stage of a project until fault tolerance and scalability
are necessary. Storer enables you transparently swap the backend to DBS like [tikv](https://tikv.org/) or Postgres.
Currently, badger and Postgres are supported. It's fairly easy to create new adapters, only need to implement 5 functions.
## Features
- Manipulate records like normal list items in golang
- Complex indexing, such as compound indexes, object index, etc
- Transactions between collections and indexes
- No database is perfect, use whatever backend that fits, default is [badger](https://github.com/dgraph-io/badger)
- Graceful schema migration on the fly, no more stop the world migration, [how it works](pkg/typee/README.md)
## Examples
Check the [example file](examples_test.go). Also the [examples folder](examples).
```go
package main
import "github.com/ysmood/storer"
type User struct {
Name string
Level int
}
func main() {
store := storer.New("db-dir")
users := store.List(&User{})
// setup index
index := users.Index("level", func(u *User) interface{} {
return u.Level % 2
})
// add
users.Add(&User{"Lee", 1})
users.Add(&User{"Maa", 2})
users.Add(&User{"Ann", 4})
// get all even level users
var even []User
index.From(0).Find(&even)
// here we should get Maa and Ann
}
```
## Benchmarks
This lib only added 3 layers above the underlying backend:
- minimum key prefixing, normally one byte per key, the algorithm is [here](https://github.com/ysmood/byframe)
- minimum data encoding, the encoding used for benchmarking is [msgpack](https://github.com/vmihailenco/msgpack)
- index items with an extra record, so need at least two gets to retrieve an item via its index
So theoretically, the performance should be nearly the same with bare get and set when data is small.
Run `go test -bench Benchmark -benchmem`, here's a sample output:
```txt
BenchmarkBadgerSet-6 20000 72780 ns/op 2571 B/op 81 allocs/op
BenchmarkSet-6 20000 74056 ns/op 2755 B/op 95 allocs/op
BenchmarkSetWithIndex-6 20000 79945 ns/op 3666 B/op 134 allocs/op
BenchmarkBadgerGet-6 1000000 1392 ns/op 528 B/op 13 allocs/op
BenchmarkGet-6 1000000 1769 ns/op 624 B/op 19 allocs/op
BenchmarkBadgerPrefixGet-6 500000 3672 ns/op 1552 B/op 34 allocs/op
BenchmarkGetByIndex-6 300000 4615 ns/op 1792 B/op 46 allocs/op
BenchmarkFilter-6 50000 23635 ns/op 7742 B/op 185 allocs/op
```
The ones named with badger are using badger direct to manipulate data, the ones after each badger benchmark
are the treatment group.
The benchmark shows badger's prefix-get has huge overhead over direct-get which doesn't make sense to me yet.
Overall the result is as expected, it shows the overhead of the key prefixing and data encoding decoding
can be ignored when compared with disk IO overhead.
## Development
Check the settings of Github Actions for testing.
The sugar files can be removed, it doesn't affect the core functions.