Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/recoilme/pudge
Fast and simple key/value store written using Go's standard library
https://github.com/recoilme/pudge
Last synced: 7 days ago
JSON representation
Fast and simple key/value store written using Go's standard library
- Host: GitHub
- URL: https://github.com/recoilme/pudge
- Owner: recoilme
- License: mit
- Created: 2018-11-20T10:11:53.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2021-07-04T02:08:38.000Z (over 3 years ago)
- Last Synced: 2024-07-31T20:49:31.288Z (5 months ago)
- Language: Go
- Homepage:
- Size: 70.3 KB
- Stars: 367
- Watchers: 14
- Forks: 35
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-go - pudge - Fast and simple key/value store written using Go's standard library. (Database / Databases Implemented in Go)
- awesome-go-extra - pudge - 11-20T10:11:53Z|2021-07-04T02:08:38Z| (Generators / Databases Implemented in Go)
README
[](https://godoc.org/github.com/recoilme/pudge)
[](https://goreportcard.com/report/github.com/recoilme/pudge)
[](https://travis-ci.org/recoilme/pudge)
[](https://github.com/avelino/awesome-go)
Table of Contents
=================
* [Description](#description)
* [Usage](#usage)
* [Cookbook](#cookbook)
* [Disadvantages](#disadvantages)
* [Motivation](#motivation)
* [Benchmarks](#benchmarks)
* [Test 1](#test-1)
* [Test 4](#test-4)
## Description
Package pudge is a fast and simple key/value store written using Go's standard library.
It presents the following:
* Supporting very efficient lookup, insertions and deletions
* Performance is comparable to hash tables
* Ability to get the data in sorted order, which enables additional operations like range scan
* Select with limit/offset/from key, with ordering or by prefix
* Safe for use in goroutines
* Space efficient
* Very short and simple codebase
* Well tested, used in production
## Usage
```golang
package main
import (
"log"
"github.com/recoilme/pudge"
)
func main() {
// Close all database on exit
defer pudge.CloseAll()
// Set (directories will be created)
pudge.Set("../test/test", "Hello", "World")
// Get (lazy open db if needed)
output := ""
pudge.Get("../test/test", "Hello", &output)
log.Println("Output:", output)
ExampleSelect()
}
//ExampleSelect
func ExampleSelect() {
cfg := &pudge.Config{
SyncInterval: 1} // every second fsync
db, err := pudge.Open("../test/db", cfg)
if err != nil {
log.Panic(err)
}
defer db.DeleteFile()
type Point struct {
X int
Y int
}
for i := 100; i >= 0; i-- {
p := &Point{X: i, Y: i}
db.Set(i, p)
}
var point Point
db.Get(8, &point)
log.Println(point)
// Output: {8 8}
// Select 2 keys, from 7 in ascending order
keys, _ := db.Keys(7, 2, 0, true)
for _, key := range keys {
var p Point
db.Get(key, &p)
log.Println(p)
}
// Output: {8 8}
// Output: {9 9}
}
```
## Cookbook
- Store data of any type. Pudge uses Gob encoder/decoder internally. No limits on keys/values size.
```golang
pudge.Set("strings", "Hello", "World")
pudge.Set("numbers", 1, 42)
type User struct {
Id int
Name string
}
u := &User{Id: 1, Name: "name"}
pudge.Set("users", u.Id, u)
```
- Pudge is stateless and safe for use in goroutines. You don't need to create/open files before use. Just write data to pudge, don't worry about state. [web server example](https://github.com/recoilme/pixel)
- Pudge is parallel. Readers don't block readers, but a writer - does, but by the stateless nature of pudge it's safe to use multiples files for storages.
- Default store system: like memcache + file storage. Pudge uses in-memory hashmap for keys, and writes values to files (no value data stored in memory). But you may use inmemory mode for values, with custom config:
```golang
cfg = pudge.DefaultConfig()
cfg.StoreMode = 2
db, err := pudge.Open(dbPrefix+"/"+group, cfg)
...
db.Counter(key, val)
```
In that case, all data is stored in memory and will be stored on disk only on Close.
[Example server for highload, with http api](https://github.com/recoilme/bandit-server)
- You may use pudge as an engine for creating databases.
[Example database](https://github.com/recoilme/slowpoke)
- Don't forget to close all opened databases on shutdown/kill.
```golang
// Wait for interrupt signal to gracefully shutdown the server
quit := make(chan os.Signal)
signal.Notify(quit, os.Interrupt, os.Kill)
<-quit
log.Println("Shutdown Server ...")
if err := pudge.CloseAll(); err != nil {
log.Println("Pudge Shutdown err:", err)
}
```
[example recovery function for gin framework](https://github.com/recoilme/bandit-server/blob/02e6eb9f89913bd68952ec35f6c37fc203d71fc2/bandit-server.go#L89)
- Pudge has a primitive select/query engine.
```golang
// Select 2 keys, from 7 in ascending order
keys, _ := db.Keys(7, 2, 0, true)
// select keys from db where key>7 order by keys asc limit 2 offset 0
```
- Pudge will work well on SSD or spined disks. Pudge doesn't eat memory or storage or your sandwich. No hidden compaction/rebalancing/resizing and so on tasks. No LSM Tree. No MMap. It's a very simple database with less than 500 LOC. It's good for [simple social network](https://github.com/recoilme/tgram) or highload system
## Disadvantages
- No transaction system. All operations are isolated, but you don't may batching them with automatic rollback.
- [Keys](https://godoc.org/github.com/recoilme/pudge#Keys) function (select/query engine) may be slow. Speed of query may vary from 10ms to 1sec per million keys. Pudge don't use BTree/Skiplist or Adaptive radix tree for store keys in ordered way on every insert. Ordering operation is "lazy" and run only if needed.
- If you need storage or database for hundreds of millions keys - take a look at [Sniper](https://github.com/recoilme/sniper) or [b52](https://github.com/recoilme/b52). They are optimized for highload (pudge - not).
- No fsync on every insert. Most of database fsync data by the timer too
- Deleted data don't remove from physically (but upsert will try to reuse space). You may shrink database only with backup right now
```golang
pudge.BackupAll("backup")
```
- Keys automatically convert to binary and ordered with binary comparator. It's simple for use, but ordering will not work correctly for negative numbers for example
- Author of project don't work at Google or Facebook and his name not Howard Chu or Brad Fitzpatrick. But I'm open for issue or contributions.
## Motivation
Some databases very well for writing. Some of the databases very well for reading. But [pudge is well balanced for both types of operations](https://github.com/recoilme/pogreb-bench). It has small [cute api](https://godoc.org/github.com/recoilme/pudge), and don't have hidden graveyards. It's just hashmap where values written in files. And you may use one database for in-memory/persistent storage in a stateless stressfree way
## Benchmarks
[All tests here](https://github.com/recoilme/pogreb-bench)
***Some tests, MacBook Pro (Retina, 13-inch, Early 2015)***
### Test 1
Number of keys: 1000000
Minimum key size: 16, maximum key size: 64
Minimum value size: 128, maximum value size: 512
Concurrency: 2
| | pogreb | goleveldb | bolt | badgerdb | pudge | slowpoke | pudge(mem) |
|-----------------------|---------|-----------|--------|----------|--------|----------|------------|
| 1M (Put+Get), seconds | 187 | 38 | 126 | 34 | 23 | 23 | 2 |
| 1M Put, ops/sec | 5336 | 34743 | 8054 | 33539 | 47298 | 46789 | 439581 |
| 1M Get, ops/sec | 1782423 | 98406 | 499871 | 220597 | 499172 | 445783 | 1652069 |
| FileSize,Mb | 568 | 357 | 552 | 487 | 358 | 358 | 358 |
### Test 4
Number of keys: 10000000
Key size: 8
Value size: 16
Concurrency: 100
| | goleveldb | badgerdb | pudge |
|-----------------------|-----------|----------|--------|
| 10M (Put+Get), seconds| 165 | 120 | 243 |
| 10M Put, ops/sec | 122933 | 135709 | 43843 |
| 10M Get, ops/sec | 118722 | 214981 | 666067 |
| FileSize,Mb | 312 | 1370 | 381 |