{"id":13614340,"url":"https://github.com/douban/gobeansdb","last_synced_at":"2025-04-04T16:16:15.400Z","repository":{"id":47042087,"uuid":"143987550","full_name":"douban/gobeansdb","owner":"douban","description":"Distributed object storage server from Douban Inc. ","archived":false,"fork":false,"pushed_at":"2024-04-02T10:15:38.000Z","size":3352,"stargazers_count":450,"open_issues_count":3,"forks_count":63,"subscribers_count":22,"default_branch":"master","last_synced_at":"2025-03-28T15:08:26.373Z","etag":null,"topics":["distributed-storage","kvstore","memcached","object-storage"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/douban.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-08-08T08:56:34.000Z","updated_at":"2025-01-24T11:38:37.000Z","dependencies_parsed_at":"2022-09-26T16:20:37.491Z","dependency_job_id":"73b1ba9e-d178-4b48-b081-8de188a9e506","html_url":"https://github.com/douban/gobeansdb","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/douban%2Fgobeansdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/douban%2Fgobeansdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/douban%2Fgobeansdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/douban%2Fgobeansdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/douban","download_url":"https://codeload.github.com/douban/gobeansdb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247208181,"owners_count":20901570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-storage","kvstore","memcached","object-storage"],"created_at":"2024-08-01T20:01:00.324Z","updated_at":"2025-04-04T16:16:15.371Z","avatar_url":"https://github.com/douban.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"\n# GoBeansDB ![](https://github.com/douban/gobeansdb/workflows/GoBeansDB%20Test/badge.svg) [![Release](https://img.shields.io/github/v/release/douban/gobeansdb)](https://github.com/douban/gobeansdb/releases)\n\nYet another distributed key-value storage system from Douban Inc.\n\nAny memcached client cache interactive with GobeansDB without any modification.\n\n## Related\n\n- [libmc](https://github.com/douban/libmc) : a high performance python/go mc client\n- [gobeansproxy](https://github.com/douban/gobeansproxy) : routing to gobeansdb cluster with three copy\n- [beansdbadmin](https://github.com/douban/beansdbadmin): webUI, sync ...\n\n## Prepare\n\nGoBeansDB uses `go mod` to manage dependencies, please make sure your Go version \u003e= 1.11.0 first.\n\n\n## Install\n\n```shell\n$ git clone http://github.com/douban/gobeansdb.git\n$ cd gobeansdb\n$ go mod vendor\n$ make\n```\n\n## test\n\n```shell\n$ make test  # unit test\n$ make pytest  # Integrated test\n```\n\n## run\n\n```shell\n$ ${GOPATH}/bin/gobeansdb -h\n```\n\n## Python Example\n\n```\nimport libmc\n\n\nmc = libmc.Client(['localhost:7900'])\nmc.set(\"foo\", \"bar\")\nmc.get(\"foo\")\n\n```\n\n## Features\n\n- 协议： memcached。推荐 libmc 客户端（c++ 实现，目前支持 go 和 python，基于 poll 的并发 get_multi/set_multi）\n- sharding： 静态 hash 路由，分桶数 16 整数倍\n- 索引： 内存索引全部 key，开销约为每个 key 20 字节，主要内存用 c 分配。 get hit 直接定位到文件 record，get miss 不落盘。\n- 最终一致性：同步脚本不断比较一个桶三副本 htree（每个桶一个 16 叉的内存 merkle tree）做同步，比较时间戳。\n- 文件格式：data 文件可以看成 log（顺序写入）； 每个 record 256 bytes 对齐，有 crc 校验。\n\n## 在 douban 使用方法\n\n```\nmc_client --- cache\n          |\n          --- any beansdb proxy -- beansdb servers \n```\n\n- 用于单个 key 并发写很少的数据\n- [gobeansproxy](https://github.com/douban/gobeansproxy) 负责路由， 固定 3 副本\n- 两个集群： 分别存储 图片类 （cache 为 CDN）  和 长文本 （cache 为 mc 集群）。\n- 支持离线  [dpark](https://github.com/douban/dpark) 读写，读支持本地化。\n- 可以视情况保留一段时间完整的写记录。\n- 借助 python 脚本 管理，近期整理后会部分开源，包 admin UI（readonly），同步脚本等\n\n\n磁盘上的样子（256分区）：\n\n* /var/lib/beansdb\n\t* 0/\n\t\t* 0/  -\u003e /data1/beansdb/0/0\n\t\t\t* 000.data\n\t\t\t* 000.000.idx.s\n\t\t\t* 000.001.idx.s\n\t\t\t* ...\n\t\t\t* 008.000.idx.hash\n\t\t\t* ...\n\t\t\t* 009.data\n\t\t\t* 009.000.idx.s\t\t\n\n\n## 入坑指南\n\n优点\n\n1. 数据文件即 log， 结构简单，数据安全\n2. htree 设计 方便数据同步；\n3. 全内存索引，访问磁盘次数少，尤其可以准确过滤掉不存在的 key。\n\n缺点/注意\n\n1. 一致性支持较弱，时间戳目前是秒级（受限于数据文件格式）。\n2. 全内存索引，有一定内存开销，在启动时载入索引略慢（约十几秒到半分钟， 决定于key 数量）。\n3. 数据文件格式的 padding 对小 value 有一定浪费。\n\n\n配置重点（详见 wiki）\n\n- 分桶数\n- htree 高度（决定 merkle tree 部分内存大小和 溢出链表的平均长度）\n\n\n## 与 [beansdb](https://github.com/douban/beansdb) 关系\n\n- 兼容\n  - 数据文件格式不变\n  - 仍使用 mc 协议（\"@\" \"?\" 开头的特殊 key 用法略不同）\n  - htree 的核心设计不变\n- 改进\n  - go 实现，更稳定，方便配置和维护（http api等）\n  - 内存开销变小\n  - hint 文件格式和后缀变了\n  - 同一个节点不同数据文件不在混部署，以避免坏一个损失整个点数据\n\n\n# 一些设计/实现要点见 [wiki](https://github.com/douban/gobeansdb/wiki)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdouban%2Fgobeansdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdouban%2Fgobeansdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdouban%2Fgobeansdb/lists"}