{"id":13612627,"url":"https://github.com/rpcxio/basalt","last_synced_at":"2026-02-03T09:10:41.504Z","repository":{"id":115963627,"uuid":"240256442","full_name":"rpcxio/basalt","owner":"rpcxio","description":"高性能的分布式的专门空间优化的 Bitmap 服务, 高效检查数据是否存在,日活统计，签到，打点等等","archived":false,"fork":false,"pushed_at":"2020-08-13T04:00:31.000Z","size":330,"stargazers_count":189,"open_issues_count":1,"forks_count":23,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-12-17T14:53:12.693Z","etag":null,"topics":["bitmap","bloom-filter","cuckoo-filter","microservices","roaring-bitmaps"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rpcxio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-02-13T12:35:23.000Z","updated_at":"2025-11-05T09:07:37.000Z","dependencies_parsed_at":null,"dependency_job_id":"ad776367-f3d1-45af-b815-af08686260f6","html_url":"https://github.com/rpcxio/basalt","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/rpcxio/basalt","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rpcxio%2Fbasalt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rpcxio%2Fbasalt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rpcxio%2Fbasalt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rpcxio%2Fbasalt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rpcxio","download_url":"https://codeload.github.com/rpcxio/basalt/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rpcxio%2Fbasalt/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29039342,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-03T08:41:49.363Z","status":"ssl_error","status_checked_at":"2026-02-03T08:40:19.255Z","response_time":96,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bitmap","bloom-filter","cuckoo-filter","microservices","roaring-bitmaps"],"created_at":"2024-08-01T20:00:32.510Z","updated_at":"2026-02-03T09:10:41.480Z","avatar_url":"https://github.com/rpcxio.png","language":"Go","readme":"# 基于Raft的数据一致性分布式的 bitmap 服务\n\n`bitmap`(位图)技术是数据库、大数据和互联网业务等场景下经常使用的一种技术。\n\n- 存在性判断\n  - 爬虫url去重\n  - 垃圾邮件过滤\n  - 用户已阅读\n  - 用户已赞\n  - ...\n- 去重\n- 数据库\n- 大数据计算\n\n![](examples/basalt.jpg)\n\n## 架构图\n\n![](doc/architecture.png)\n\n支持三种协议读写: `HTTP`、`redcis`和`rpcx`。\n\n`写`操作可以发给任意节点，但最终是由raft leader节点进行操作。\n`读`操作可以选择任意的节点进行读取。\n\n## 服务\n\n进入`cmd/server`, 运行`go run server.go`启动一个bitmap服务。\n\n它同时支持三种服务:\n\n- rpcx: 你可以使用rpcx服务获取高性能的网络调用, `cmd/rpcx_client`是一个rpcx客户端的demo\n- redis: 你可以使用redis客户端访问Bitmap服务(如果你的redis client支持自定义命令), 方便兼容redis调用代码， `cmd/redis_client`是redis demo\n- http: 通过http服务调用，调用简单,支持各种编程语言和脚本，`cmd/http_client/curl.sh`是通过`curl`调用服务\n\n## 集群模式\n\n支持raft集群模式: [basalt集群](https://github.com/rpcxio/basalt/tree/master/cmd/raft_server)\n\n## API接口\n\nbasalt位图服务支持三种接口模式：\n\n- HTTP API: 通过http api的方式进行访问\n- Redis模式: 扩展了redis命令，可以通过redis client进行访问\n- rpcx模式: 可以通过rpcx框架进行访问\n\n### Redis命令\n\n- `ping`: ping-pong消息\n- `quit`: 退出连接\n- `bmadd name value`: 在名为`name`的bitmap增加一个uint32值`value`\n- `bmaddmany name value1 value2 value3...`: 为名为`name`的bitmap增加一批值\n- `bmdel name value`: 在名为`name`的bitmap删除一个uint32值`value`\n- `bmdrop name`: 删除名为`name`的bitmap\n- `bmclear name`: 清空名为`name`的bitmap\n- `bmcard name`: 获取为`name`的bitmap包含的元素数\n- `bmexists name value`: 检查uint32值`value`是否存在于名为`name`的bitmap中，整数`1`代表存在，`0`代表不存在\n- `bminter name1 name2 name3...`: 求几个bitmap的交集，返回交集的uint32整数列表\n- `bminterstore dst name1 name2 name3...`: 求几个bitmap(`name1`、`name2`、`name3`...)的交集，并将结果保存到`dst`中\n- `bmunion name1 name2 name3...`: 求几个bitmap的并集，返回并集的uint32整数列表\n- `bmunionstore dst name1 name2 name3...`: 求几个bitmap(`name1`、`name2`、`name3`...)的并集，并将结果保存到`dst`中\n- `bmxor name1 name2.`: 求两个bitmap的`xor`集(双方不共有的集合,相当于并集减交集)，返回`xor`集的uint32整数列表\n- `bmxorstore dst name1 name2`: 求两个bitmap的`xor`集，并将结果保存到`dst`中\n- `bmdiff name1 name2`: 求`name1`中和`name2`没有交集的数据，返回结果的uint32整数列表\n- `bmdiffstore dst name1 name2`: 求`name1`中和`name2`没有交集的数据，并将结果保存到`dst`中\n- `bmstats name`: 返回`name`的bitmap的统计信息\n\n### rpcx 服务\n\n查看 [godoc](https://godoc.org/github.com/rpcxio/basalt)以了解提供的rpcx服务\n\n### HTTP 服务\n\nHTTP 服务提供和 redis、rpcx服务相同的功能，通过http调用就可以访问Bitmap服务。\n\n所有的参数都是在路径中提供，路径格式为`/action/param1/param2`。\n复数形式`values`、`names`包含多个元素，元素以逗号`,`分隔。\n\n为了简化操作，所有的http服务都是通过`GET`方法提供的。\n\n返回的`HTTP StatusCode`代表的含义如下：\n\n- `200` 代表`OK`、`存在`\n- `400` 代表参数不对，比如应该是uint32格式，结果却是无法解析的字符串\n- `404` 代表不存在\n- `500` 代表内部处理错误\n\n\nHTTP服务路径列表如下：\n\n- `/add/:name/:value`\n- `/addmany/:name/:values`\n- `/remove/:name/:value`\n- `/drop/:name`\n- `/clear/:name`\n- `/exists/:name/:value`\n- `/card/:name`\n- `/inter/:names`\n- `/interstore/:dst/:names`\n- `/union/:names`\n- `/unionstore/:dst/:names`\n- `/xor/:name1/:name2`\n- `/xorstore/:dst/:name1/:name2`\n- `/diff/:name1/:name2`\n- `/diffstore/:dst/:name1/:name2`\n- `/stats/:name`\n\n## 例子\n\n以微博关注关系数据集做例子，我们使用Bitmap服务来存储某人是否关注了某人，以及两人是否互相关注。\n\n例子参看 [weibo_follow](https://github.com/rpcxio/basalt/tree/master/examples/weibo)\n\n\n## Roadmap\n\n- [x] Multiple-key Bitmap\n- [x] rpcx services for Bitmap\n- [x] HTTP services for Bitmap\n- [x] Redis services for Bitmap\n- [x] Persistence\n- [x] Cluster mode\n\n## Credits\n\n- [roaring](https://github.com/RoaringBitmap/roaring)\n- [redcon](https://github.com/tidwall/redcon)\n- [httprouter](https://github.com/julienschmidt/httprouter)\n- [rpcx](https://github.com/smallnest/rpcx)\n- [etcd](https://github.com/etcd-io/etcd)","funding_links":[],"categories":["开源类库","Open source library","Go"],"sub_categories":["算法","Algorithm"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frpcxio%2Fbasalt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frpcxio%2Fbasalt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frpcxio%2Fbasalt/lists"}