Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/recoilme/slowpoke
Low-level key/value store in pure Go.
https://github.com/recoilme/slowpoke
database embedded engine golang key-value persistence
Last synced: 3 months ago
JSON representation
Low-level key/value store in pure Go.
- Host: GitHub
- URL: https://github.com/recoilme/slowpoke
- Owner: recoilme
- License: mit
- Archived: true
- Created: 2018-02-19T09:22:37.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2019-09-30T09:10:54.000Z (about 5 years ago)
- Last Synced: 2024-07-13T13:41:48.492Z (4 months ago)
- Topics: database, embedded, engine, golang, key-value, persistence
- Language: Go
- Homepage:
- Size: 2.23 MB
- Stars: 100
- Watchers: 8
- Forks: 9
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-go - slowpoke - Package slowpoke implements a low-level key/value store in pure Go. Keys stored in memory (with persistence), values stored on disk. - ★ 65 (Database)
README
[](https://travis-ci.org/recoilme/slowpoke) [](https://goreportcard.com/report/github.com/recoilme/slowpoke)
[](https://godoc.org/github.com/recoilme/slowpoke)
**Description**
Package slowpoke is a simple key/value store written using Go's standard library only. Keys are stored in memory (with persistence), values stored on disk.
Description on russian: https://habr.com/post/354224/
**Motivation**
Replace [Bolt](https://github.com/boltdb/bolt) with a simpler and more efficient engine.
Slowpoke (from version 2.0) based on [pudge](https://github.com/recoilme/pudge)
**How it works**
Keys are stored in memory with persistence to disk. Values stored on disk only.
**Slowpoke is parallel**
**Server**
GRPC Server example: [okdb](https://github.com/recoilme/okdb)
**Complex examples**
[typegram](https://github.com/recoilme/tgram)
zen platform for authors and their subscribers with a minimalistic design and user-friendly interface.
[golang-gin-realworld-example-app](https://github.com/recoilme/golang-gin-realworld-example-app)
This codebase was created to demonstrate a fully fledged fullstack application built with Golang/Gin/Slowpoke including CRUD operations, authentication, routing, pagination, and more.
**Basic example**
```golang
package main
import (
"fmt"
"github.com/recoilme/slowpoke"
)
func main() {
// create database
file := "test/example.db"
// close all opened database
defer slowpoke.CloseAll()
// init key/val
key := []byte("foo")
val := []byte("bar")
//store
slowpoke.Set(file, key, val)
// get
res, _ := slowpoke.Get(file, key)
//result
fmt.Println(string(res))
}
```
**Lazy example**
```golang
func TestGob(t *testing.T) {
file := "test/gob.db"
DeleteFile(file)
defer CloseAll()
type Post struct {
Id int
Content string
Category string
}
for i := 0; i < 20; i++ {
post := &Post{Id: i, Content: "Content:" + strconv.Itoa(i)}
err := SetGob(file, post.Id, post)
ch(err, t)
}
for i := 0; i < 20; i++ {
var post = new(Post)
err := GetGob(file, i, post)
ch(err, t)
fmt.Println("i:", i, "Post:", post)
}
}
```
**Advanced example**
```golang
type Post struct {
Id int
Content string
Category string
}
func main() {
posts := "test/posts"
tags := "test/tags"
var pairs [][]byte
for i := 0; i < 40; i++ {
id := make([]byte, 4)
binary.BigEndian.PutUint32(id, uint32(i))
post := &Post{Id: i, Content: "Content:" + strconv.Itoa(i), Category: "Category:" + strconv.Itoa(i/10)}
b, _ := json.Marshal(post)
pairs = append(pairs, id)
pairs = append(pairs, b)
tag := fmt.Sprintf("%s:%08d", strconv.Itoa(i/10), i)
//store only tags keys
slowpoke.Set(tags, []byte(tag), nil)
}
//store posts fast
slowpoke.Sets(posts, pairs)
//get last 2 post key with offset 2
limit := uint32(2)
offset := uint32(2)
order := false //desc
keys, _ := slowpoke.Keys(posts, nil, limit, offset, order)
fmt.Println(keys) //[[0 0 0 37] [0 0 0 36]]
//get key/ values
res := slowpoke.Gets(posts, keys)
for k, v := range res {
if k%2 == 0 {
fmt.Print(binary.BigEndian.Uint32(v))
} else {
var p Post
json.Unmarshal(v, &p)
fmt.Println(p)
}
}
//37{37 Content:37 Category:3}
//36{36 Content:36 Category:3}
//free from memory
slowpoke.Close(posts)
slowpoke.Close(tags)
//open Db and read tags by prefix 2:* in ascending order
tagsKeys, _ := slowpoke.Keys(tags, []byte("2:*"), 0, 0, true)
for _, v := range tagsKeys {
fmt.Print(string(v) + ", ")
}
//2:00000020, 2:00000021, 2:00000022, 2:00000023, 2:00000024, 2:00000025, 2:00000026, 2:00000027, 2:00000028, 2:00000029,
}
```
**Api**
All methods are thread-safe.
- **Set/Sets/SetGob**
Store val and key. If the file does not exist it will be created.
- **Get/Gets/GetGob**
Return the value for the given key or nil and an error. `Get` will open the database if necessary.
- **Keys**
Return keys in ascending/descending order.
With limit and offset.
If `from` is not nil, return keys lexicographically greater than the `from` value.
If `from` ends with asterix `*`, return keys with the prefix equal to `from` without the asterix.
[Documentation](https://godoc.org/github.com/recoilme/slowpoke)
**Status**
Used in production (master branch)
**Benchmark**
[All tests here](https://github.com/recoilme/pogreb-bench)
Some tests, MacBook Pro (Retina, 13-inch, Early 2015)
### Test 1
Number of keys: 1000000
Minimum key size: 16, maximum key size: 64
Minimum value size: 128, maximum value size: 512
Concurrency: 2
| | pogreb | goleveldb | bolt | badgerdb | pudge | slowpoke | pudge(mem) |
|-----------------------|---------|-----------|--------|----------|--------|----------|------------|
| 1M (Put+Get), seconds | 187 | 38 | 126 | 34 | 23 | 23 | 2 |
| 1M Put, ops/sec | 5336 | 34743 | 8054 | 33539 | 47298 | 46789 | 439581 |
| 1M Get, ops/sec | 1782423 | 98406 | 499871 | 220597 | 499172 | 445783 | 1652069 |
| FileSize,Mb | 568 | 357 | 552 | 487 | 358 | 358 | 358 |