{"id":13411887,"url":"https://github.com/peterbourgon/diskv","last_synced_at":"2025-05-13T23:04:02.156Z","repository":{"id":2788919,"uuid":"3788841","full_name":"peterbourgon/diskv","owner":"peterbourgon","description":"A disk-backed key-value store.","archived":false,"fork":false,"pushed_at":"2021-11-10T01:12:08.000Z","size":138,"stargazers_count":1431,"open_issues_count":8,"forks_count":104,"subscribers_count":40,"default_branch":"master","last_synced_at":"2025-04-30T07:09:02.461Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://godoc.org/github.com/peterbourgon/diskv","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/peterbourgon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-03-21T16:44:32.000Z","updated_at":"2025-04-21T10:16:40.000Z","dependencies_parsed_at":"2022-07-09T08:16:23.148Z","dependency_job_id":null,"html_url":"https://github.com/peterbourgon/diskv","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peterbourgon%2Fdiskv","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peterbourgon%2Fdiskv/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peterbourgon%2Fdiskv/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peterbourgon%2Fdiskv/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/peterbourgon","download_url":"https://codeload.github.com/peterbourgon/diskv/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254040415,"owners_count":22004533,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T20:01:18.037Z","updated_at":"2025-05-13T23:04:02.132Z","avatar_url":"https://github.com/peterbourgon.png","language":"Go","readme":"# What is diskv?\n\nDiskv (disk-vee) is a simple, persistent key-value store written in the Go\nlanguage. It starts with an incredibly simple API for storing arbitrary data on\na filesystem by key, and builds several layers of performance-enhancing\nabstraction on top.  The end result is a conceptually simple, but highly\nperformant, disk-backed storage system.\n\n[![Build Status][1]][2]\n\n[1]: https://drone.io/github.com/peterbourgon/diskv/status.png\n[2]: https://drone.io/github.com/peterbourgon/diskv/latest\n\n\n# Installing\n\nInstall [Go 1][3], either [from source][4] or [with a prepackaged binary][5].\nThen,\n\n```bash\n$ go get github.com/peterbourgon/diskv/v3\n```\n\n[3]: http://golang.org\n[4]: http://golang.org/doc/install/source\n[5]: http://golang.org/doc/install\n\n\n# Usage\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"github.com/peterbourgon/diskv/v3\"\n)\n\nfunc main() {\n\t// Simplest transform function: put all the data files into the base dir.\n\tflatTransform := func(s string) []string { return []string{} }\n\n\t// Initialize a new diskv store, rooted at \"my-data-dir\", with a 1MB cache.\n\td := diskv.New(diskv.Options{\n\t\tBasePath:     \"my-data-dir\",\n\t\tTransform:    flatTransform,\n\t\tCacheSizeMax: 1024 * 1024,\n\t})\n\n\t// Write three bytes to the key \"alpha\".\n\tkey := \"alpha\"\n\td.Write(key, []byte{'1', '2', '3'})\n\n\t// Read the value back out of the store.\n\tvalue, _ := d.Read(key)\n\tfmt.Printf(\"%v\\n\", value)\n\n\t// Erase the key+value from the store (and the disk).\n\td.Erase(key)\n}\n```\n\nMore complex examples can be found in the \"examples\" subdirectory.\n\n\n# Theory\n\n## Basic idea\n\nAt its core, diskv is a map of a key (`string`) to arbitrary data (`[]byte`).\nThe data is written to a single file on disk, with the same name as the key.\nThe key determines where that file will be stored, via a user-provided\n`TransformFunc`, which takes a key and returns a slice (`[]string`)\ncorresponding to a path list where the key file will be stored. The simplest\nTransformFunc,\n\n```go\nfunc SimpleTransform (key string) []string {\n    return []string{}\n}\n```\n\nwill place all keys in the same, base directory. The design is inspired by\n[Redis diskstore][6]; a TransformFunc which emulates the default diskstore\nbehavior is available in the content-addressable-storage example.\n\n[6]: http://groups.google.com/group/redis-db/browse_thread/thread/d444bc786689bde9?pli=1\n\n**Note** that your TransformFunc should ensure that one valid key doesn't\ntransform to a subset of another valid key. That is, it shouldn't be possible\nto construct valid keys that resolve to directory names. As a concrete example,\nif your TransformFunc splits on every 3 characters, then\n\n```go\nd.Write(\"abcabc\", val) // OK: written to \u003cbase\u003e/abc/abc/abcabc\nd.Write(\"abc\", val)    // Error: attempted write to \u003cbase\u003e/abc/abc, but it's a directory\n```\n\nThis will be addressed in an upcoming version of diskv.\n\nProbably the most important design principle behind diskv is that your data is\nalways flatly available on the disk. diskv will never do anything that would\nprevent you from accessing, copying, backing up, or otherwise interacting with\nyour data via common UNIX commandline tools.\n\n## Advanced path transformation\n\nIf you need more control over the file name written to disk or if you want to support\nslashes in your key name or special characters in the keys, you can use the\nAdvancedTransform property. You must supply a function that returns\na special PathKey structure, which is a breakdown of a path and a file name. Strings\nreturned must be clean of any slashes or special characters:\n\n```go\nfunc AdvancedTransformExample(key string) *diskv.PathKey {\n\tpath := strings.Split(key, \"/\")\n\tlast := len(path) - 1\n\treturn \u0026diskv.PathKey{\n\t\tPath:     path[:last],\n\t\tFileName: path[last] + \".txt\",\n\t}\n}\n\n// If you provide an AdvancedTransform, you must also provide its\n// inverse:\n\nfunc InverseTransformExample(pathKey *diskv.PathKey) (key string) {\n\ttxt := pathKey.FileName[len(pathKey.FileName)-4:]\n\tif txt != \".txt\" {\n\t\tpanic(\"Invalid file found in storage folder!\")\n\t}\n\treturn strings.Join(pathKey.Path, \"/\") + pathKey.FileName[:len(pathKey.FileName)-4]\n}\n\nfunc main() {\n\td := diskv.New(diskv.Options{\n\t\tBasePath:          \"my-data-dir\",\n\t\tAdvancedTransform: AdvancedTransformExample,\n\t\tInverseTransform:  InverseTransformExample,\n\t\tCacheSizeMax:      1024 * 1024,\n\t})\n\t// Write some text to the key \"alpha/beta/gamma\".\n\tkey := \"alpha/beta/gamma\"\n\td.WriteString(key, \"¡Hola!\") // will be stored in \"\u003cbasedir\u003e/alpha/beta/gamma.txt\"\n\tfmt.Println(d.ReadString(\"alpha/beta/gamma\"))\n}\n```\n\n\n## Adding a cache\n\nAn in-memory caching layer is provided by combining the BasicStore\nfunctionality with a simple map structure, and keeping it up-to-date as\nappropriate. Since the map structure in Go is not threadsafe, it's combined\nwith a RWMutex to provide safe concurrent access.\n\n## Adding order\n\ndiskv is a key-value store and therefore inherently unordered. An ordering\nsystem can be injected into the store by passing something which satisfies the\ndiskv.Index interface. (A default implementation, using Google's\n[btree][7] package, is provided.) Basically, diskv keeps an ordered (by a\nuser-provided Less function) index of the keys, which can be queried.\n\n[7]: https://github.com/google/btree\n\n## Adding compression\n\nSomething which implements the diskv.Compression interface may be passed\nduring store creation, so that all Writes and Reads are filtered through\na compression/decompression pipeline. Several default implementations,\nusing stdlib compression algorithms, are provided. Note that data is cached\ncompressed; the cost of decompression is borne with each Read.\n\n## Streaming\n\ndiskv also now provides ReadStream and WriteStream methods, to allow very large\ndata to be handled efficiently.\n\n\n# Future plans\n\n * Needs plenty of robust testing: huge datasets, etc...\n * More thorough benchmarking\n * Your suggestions for use-cases I haven't thought of\n\n\n# Credits and contributions\n\nOriginal idea, design and implementation: [Peter Bourgon](https://github.com/peterbourgon)\nOther collaborations: [Javier Peletier](https://github.com/jpeletier) ([Epic Labs](https://www.epiclabs.io))\n","funding_links":[],"categories":["Database","开源类库","Go-lang","Go","Open source library","数据库","Programming Languages","Data Integration Frameworks","数据库  `go语言实现的数据库`","Uncategorized","Key-Value","Generators","\u003cspan id=\"数据库-database\"\u003e数据库 Database\u003c/span\u003e"],"sub_categories":["Databases Implemented in Go","Advanced Console UIs","缓存","Cache","Go","Go中实现的数据库","\u003cspan id=\"高级控制台用户界面-advanced-console-uis\"\u003e高级控制台用户界面 Advanced Console UIs\u003c/span\u003e","标准 CLI"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpeterbourgon%2Fdiskv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpeterbourgon%2Fdiskv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpeterbourgon%2Fdiskv/lists"}