{"id":13411962,"url":"https://github.com/recoilme/pudge","last_synced_at":"2025-04-04T08:05:28.255Z","repository":{"id":48949509,"uuid":"158369223","full_name":"recoilme/pudge","owner":"recoilme","description":"Fast and simple key/value store written using Go's standard library","archived":false,"fork":false,"pushed_at":"2021-07-04T02:08:38.000Z","size":72,"stargazers_count":367,"open_issues_count":0,"forks_count":35,"subscribers_count":14,"default_branch":"master","last_synced_at":"2024-07-31T20:49:31.288Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/recoilme.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-11-20T10:11:53.000Z","updated_at":"2024-07-22T02:45:48.000Z","dependencies_parsed_at":"2022-08-30T09:40:52.782Z","dependency_job_id":null,"html_url":"https://github.com/recoilme/pudge","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/recoilme%2Fpudge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/recoilme%2Fpudge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/recoilme%2Fpudge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/recoilme%2Fpudge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/recoilme","download_url":"https://codeload.github.com/recoilme/pudge/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247139314,"owners_count":20890201,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T20:01:19.310Z","updated_at":"2025-04-04T08:05:28.236Z","avatar_url":"https://github.com/recoilme.png","language":"Go","funding_links":[],"categories":["Database","数据库","数据库  `go语言实现的数据库`","Generators","Data Integration Frameworks","Uncategorized"],"sub_categories":["Databases Implemented in Go","Advanced Console UIs","Go中实现的数据库","标准 CLI"],"readme":"[![Documentation](https://godoc.org/github.com/recoilme/pudge?status.svg)](https://godoc.org/github.com/recoilme/pudge)\n[![Go Report Card](https://goreportcard.com/badge/github.com/recoilme/pudge)](https://goreportcard.com/report/github.com/recoilme/pudge)\n[![Build Status](https://travis-ci.org/recoilme/pudge.svg?branch=master)](https://travis-ci.org/recoilme/pudge)\n[![Mentioned in Awesome Go](https://awesome.re/mentioned-badge-flat.svg)](https://github.com/avelino/awesome-go)\n\nTable of Contents\n=================\n\n* [Description](#description)\n* [Usage](#usage)\n* [Cookbook](#cookbook)\n* [Disadvantages](#disadvantages)\n* [Motivation](#motivation)\n* [Benchmarks](#benchmarks)\n\t* [Test 1](#test-1)\n\t* [Test 4](#test-4)\n\n## Description\n\nPackage pudge is a fast and simple key/value store written using Go's standard library.\n\nIt presents the following:\n* Supporting very efficient lookup, insertions and deletions\n* Performance is comparable to hash tables\n* Ability to get the data in sorted order, which enables additional operations like range scan\n* Select with limit/offset/from key, with ordering or by prefix\n* Safe for use in goroutines\n* Space efficient\n* Very short and simple codebase\n* Well tested, used in production\n\n![pudge](https://avatars3.githubusercontent.com/u/417177?s=460\u0026v=4)\n\n## Usage\n\n\n```golang\npackage main\n\nimport (\n\t\"log\"\n\n\t\"github.com/recoilme/pudge\"\n)\n\nfunc main() {\n\t// Close all database on exit\n\tdefer pudge.CloseAll()\n\n\t// Set (directories will be created)\n\tpudge.Set(\"../test/test\", \"Hello\", \"World\")\n\n\t// Get (lazy open db if needed)\n\toutput := \"\"\n\tpudge.Get(\"../test/test\", \"Hello\", \u0026output)\n\tlog.Println(\"Output:\", output)\n\n\tExampleSelect()\n}\n\n\n//ExampleSelect\nfunc ExampleSelect() {\n\tcfg := \u0026pudge.Config{\n\t\tSyncInterval: 1} // every second fsync\n\tdb, err := pudge.Open(\"../test/db\", cfg)\n\tif err != nil {\n\t\tlog.Panic(err)\n\t}\n\tdefer db.DeleteFile()\n\ttype Point struct {\n\t\tX int\n\t\tY int\n\t}\n\tfor i := 100; i \u003e= 0; i-- {\n\t\tp := \u0026Point{X: i, Y: i}\n\t\tdb.Set(i, p)\n\t}\n\tvar point Point\n\tdb.Get(8, \u0026point)\n\tlog.Println(point)\n\t// Output: {8 8}\n\t// Select 2 keys, from 7 in ascending order\n\tkeys, _ := db.Keys(7, 2, 0, true)\n\tfor _, key := range keys {\n\t\tvar p Point\n\t\tdb.Get(key, \u0026p)\n\t\tlog.Println(p)\n\t}\n\t// Output: {8 8}\n\t// Output: {9 9}\n}\n\n```\n\n## Cookbook\n\n - Store data of any type. Pudge uses Gob encoder/decoder internally. No limits on keys/values size.\n\n```golang\npudge.Set(\"strings\", \"Hello\", \"World\")\npudge.Set(\"numbers\", 1, 42)\n\ntype User struct {\n\tId int\n\tName string\n}\nu := \u0026User{Id: 1, Name: \"name\"}\npudge.Set(\"users\", u.Id, u)\n\n```\n - Pudge is stateless and safe for use in goroutines. You don't need to create/open files before use. Just write data to pudge, don't worry about state. [web server example](https://github.com/recoilme/pixel)\n\n - Pudge is parallel. Readers don't block readers, but a writer - does, but by the stateless nature of pudge it's safe to use multiples files for storages.\n\n ![Illustration from slowpoke (based on pudge)](https://camo.githubusercontent.com/a1b406485fa8cd52a98d820de706e3fd255941e9/68747470733a2f2f686162726173746f726167652e6f72672f776562742f79702f6f6b2f63332f79706f6b63333377702d70316a63657771346132323164693168752e706e67)\n\n\n - Default store system: like memcache + file storage. Pudge uses in-memory hashmap for keys, and writes values to files (no value data stored in memory). But you may use inmemory mode for values, with custom config:\n```golang\ncfg = pudge.DefaultConfig()\ncfg.StoreMode = 2\ndb, err := pudge.Open(dbPrefix+\"/\"+group, cfg)\n...\ndb.Counter(key, val)\n```\nIn that case, all data is stored in memory and will be stored on disk only on Close. \n\n[Example server for highload, with http api](https://github.com/recoilme/bandit-server)\n\n - You may use pudge as an engine for creating databases. \n \n [Example database](https://github.com/recoilme/slowpoke)\n\n - Don't forget to close all opened databases on shutdown/kill.\n```golang\n \t// Wait for interrupt signal to gracefully shutdown the server \n\tquit := make(chan os.Signal)\n\tsignal.Notify(quit, os.Interrupt, os.Kill)\n\t\u003c-quit\n\tlog.Println(\"Shutdown Server ...\")\n\tif err := pudge.CloseAll(); err != nil {\n\t\tlog.Println(\"Pudge Shutdown err:\", err)\n\t}\n ```\n [example recovery function for gin framework](https://github.com/recoilme/bandit-server/blob/02e6eb9f89913bd68952ec35f6c37fc203d71fc2/bandit-server.go#L89)\n\n - Pudge has a primitive select/query engine.\n ```golang\n // Select 2 keys, from 7 in ascending order\n\tkeys, _ := db.Keys(7, 2, 0, true)\n// select keys from db where key\u003e7 order by keys asc limit 2 offset 0\n ```\n\n - Pudge will work well on SSD or spined disks. Pudge doesn't eat memory or storage or your sandwich. No hidden compaction/rebalancing/resizing and so on tasks. No LSM Tree. No MMap. It's a very simple database with less than 500 LOC. It's good for [simple social network](https://github.com/recoilme/tgram) or highload system \n\n\n## Disadvantages\n\n - No transaction system. All operations are isolated, but you don't may batching them with automatic rollback.\n - [Keys](https://godoc.org/github.com/recoilme/pudge#Keys) function (select/query engine) may be slow. Speed of query may vary from 10ms to 1sec per million keys. Pudge don't use BTree/Skiplist or Adaptive radix tree for store keys in ordered way on every insert. Ordering operation is \"lazy\" and run only if needed.\n - If you need storage or database for hundreds of millions keys - take a look at [Sniper](https://github.com/recoilme/sniper) or [b52](https://github.com/recoilme/b52). They are optimized for highload (pudge - not).\n - No fsync on every insert. Most of database fsync data by the timer too\n - Deleted data don't remove from physically (but upsert will try to reuse space). You may shrink database only with backup right now\n```golang\npudge.BackupAll(\"backup\")\n```\n - Keys automatically convert to binary and ordered with binary comparator. It's simple for use, but ordering will not work correctly for negative numbers for example\n - Author of project don't work at Google or Facebook and his name not Howard Chu or Brad Fitzpatrick. But I'm open for issue or contributions.\n\n\n## Motivation\n\nSome databases very well for writing. Some of the databases very well for reading. But [pudge is well balanced for both types of operations](https://github.com/recoilme/pogreb-bench). It has small [cute api](https://godoc.org/github.com/recoilme/pudge), and don't have hidden graveyards. It's just hashmap where values written in files. And you may use one database for in-memory/persistent storage in a stateless stressfree way\n\n\n## Benchmarks\n\n[All tests here](https://github.com/recoilme/pogreb-bench)\n\n***Some tests, MacBook Pro (Retina, 13-inch, Early 2015)***\n\n\n\n### Test 1\nNumber of keys: 1000000\nMinimum key size: 16, maximum key size: 64\nMinimum value size: 128, maximum value size: 512\nConcurrency: 2\n\n\n|                       | pogreb  | goleveldb | bolt   | badgerdb | pudge  | slowpoke | pudge(mem) |\n|-----------------------|---------|-----------|--------|----------|--------|----------|------------|\n| 1M (Put+Get), seconds | 187     | 38        | 126    | 34       | 23     | 23       | 2          |\n| 1M Put, ops/sec       | 5336    | 34743     | 8054   | 33539    | 47298  | 46789    | 439581     |\n| 1M Get, ops/sec       | 1782423 | 98406     | 499871 | 220597   | 499172 | 445783   | 1652069    |\n| FileSize,Mb           | 568     | 357       | 552    | 487      | 358    | 358      | 358        |\n\n\n### Test 4\nNumber of keys: 10000000\nKey size: 8\nValue size: 16\nConcurrency: 100\n\n\n|                       | goleveldb | badgerdb | pudge  |\n|-----------------------|-----------|----------|--------|\n| 10M (Put+Get), seconds| 165       | 120      | 243    |\n| 10M Put, ops/sec      | 122933    | 135709   | 43843  |\n| 10M Get, ops/sec      | 118722    | 214981   | 666067 |\n| FileSize,Mb           | 312       | 1370     | 381    |","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frecoilme%2Fpudge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frecoilme%2Fpudge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frecoilme%2Fpudge/lists"}