{"id":13569202,"url":"https://github.com/topling/toplingdb","last_synced_at":"2026-04-02T19:26:01.088Z","repository":{"id":37803675,"uuid":"274611759","full_name":"topling/toplingdb","owner":"topling","description":"ToplingDB is a cloud native LSM Key-Value Store with searchable compression algo and distributed compaction","archived":false,"fork":false,"pushed_at":"2026-03-31T01:48:08.000Z","size":136768,"stargazers_count":1016,"open_issues_count":9,"forks_count":153,"subscribers_count":63,"default_branch":"memtable_as_log_index","last_synced_at":"2026-03-31T04:40:17.704Z","etag":null,"topics":["compaction","database","distributed-database","kvstore","nosql","rocksdb"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/topling.png","metadata":{"files":{"readme":"README-zh_cn.md","changelog":"HISTORY.md","contributing":"CONTRIBUTING.md","funding":null,"license":"COPYING","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-06-24T08:06:01.000Z","updated_at":"2026-03-31T01:47:22.000Z","dependencies_parsed_at":"2022-06-26T01:55:41.602Z","dependency_job_id":"e453ccd7-cfe4-49c0-b00a-328bd0754671","html_url":"https://github.com/topling/toplingdb","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/topling/toplingdb","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/topling%2Ftoplingdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/topling%2Ftoplingdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/topling%2Ftoplingdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/topling%2Ftoplingdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/topling","download_url":"https://codeload.github.com/topling/toplingdb/tar.gz/refs/heads/memtable_as_log_index","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/topling%2Ftoplingdb/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31314377,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-02T12:59:32.332Z","status":"ssl_error","status_checked_at":"2026-04-02T12:54:48.875Z","response_time":89,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compaction","database","distributed-database","kvstore","nosql","rocksdb"],"created_at":"2024-08-01T14:00:37.001Z","updated_at":"2026-04-02T19:26:01.079Z","avatar_url":"https://github.com/topling.png","language":"C++","funding_links":[],"categories":["C++"],"sub_categories":[],"readme":"## ToplingDB: 一个外存上的持久化 Key-Value 存储引擎\nToplingDB 由[北京拓扑岭科技有限公司](https://topling.cn)开发与维护，从 [RocksDB](https://github.com/facebook/rocksdb) 分叉而来，详情参考 [ToplingDB 分支名称约定](https://github.com/topling/toplingdb/wiki/ToplingDB-Branch-Name-Convention)。\n\n## 快速开始\nToplingDB 需要 C++17，推荐 gcc 8.3 以上，或者 clang 也行。\n\nToplingDB 比 RocksDB 快得多，您可以自己快速验证，[下载 ToplingDB 企业版](https://topling-tools.oss-cn-qingdao.aliyuncs.com/toplingdb-8.10-trail90.tgz)，或者自己编译：\n### Compile \u0026 run db_bench\n```bash\nsudo yum -y install git libaio-devel gcc-c++ gflags-devel zlib-devel bzip2-devel libcurl-devel liburing-devel snappy-devel jemalloc-devel\n#sudo apt-get update -y \u0026\u0026 sudo apt-get install -y libjemalloc-dev libaio-dev libgflags-dev zlib1g-dev libbz2-dev libcurl4-gnutls-dev liburing-dev libsnappy-dev libbz2-dev liblz4-dev libzstd-dev\ngit clone https://github.com/topling/toplingdb\ncd toplingdb\nmake -j`nproc` db_bench DEBUG_LEVEL=0\nsudo make install PREFIX=/some/path # default is /usr/local\n```\n\n下载解压或者自行编译后，运行 [db_bench.sh](db_bench.sh)(需要[端口 2011](https://github.com/topling/rockside/blob/master/sample-conf/db_bench_enterprise.yaml#L4 \"内嵌的 http web 服务使用端口 2011\"))，然后使用 ToplingDB：[原生 C++](https://github.com/topling/rockside/wiki/101 \"典型场景是从 rocksdb 迁移过来)\")，也支持 [Java](https://github.com/topling/rockside/wiki/SidePlugin-Java-Binding \"内置在本 github 仓库中\") 和 [Rust](https://github.com/topling/rust-toplingdb \"另外的专门的 github 仓库\")。\n\n\u003e 自己编译开源版时会自动下载预编译的试用版(90天) ToplingZipTable，如果下载失败，可以给 `make` 传递变量 `WITH_TOPLING_ROCKS=0` 禁用它(或[联系我们](mailto:contact@topling.cn))。\n\n## 简单介绍\nToplingDB 的子模块 **[rockside](https://github.com/topling/rockside)** 是 ToplingDB 的入口，详情参考 **[SidePlugin wiki](https://github.com/topling/rockside/wiki)**。\n\nToplingDB 兼容 RocksDB API 的同时，增加了很多非常重要的功能与改进：\n1. [SidePlugin](https://github.com/topling/rockside/wiki) 让用户可以通过 json/yaml 文件来定义 DB 配置\n1. [内嵌 Http](https://github.com/topling/rockside/wiki/WebView) 让用户可以通过 Web 查看几乎所有 DB 信息，这是 [SidePlugin](https://github.com/topling/rockside/wiki) 的一个子功能\n1. [内嵌 Http](https://github.com/topling/rockside/wiki/WebView) 让用户可以无需重启进程，[在线修改](https://github.com/topling/rockside/wiki/Online-Change-Options) 各种 db/cf 配置，包括修改 DB 元对象（例如 MemTabFactory, TableFactory, WriteBufferManager ...）\n1. 为提升性能和可扩展性而实施的很多重构与改进，例如 MemTable 的重构\n1. MemTable 可作为 WAL 的索引，消除 MemTable 到 L0 SST 的 Flush，减小写放大，对大尺寸 MemTable 很友好\n1. 对事务处理的改进，特别是 TransactionDB 中 Lock 的管理，热点代码有 5x 以上的性能提升\n1. MultiGet 中使用 fiber/coroutine + io_uring 实现了并发 IO，比 RocksDB 自身的异步 MultiGet 又快又简洁，相应的代码量要少 100 倍不止\n1. [去虚拟化](https://github.com/topling/rockside/wiki/Devirtualization-And-Key-Prefix-Cache-Principle)，消除热点代码中的虚函数调用（主要是 Comparator），并且增加了 Key 前缀缓存，参考相应 [bechmarks](https://github.com/topling/rockside/wiki/Devirtualization-And-Key-Prefix-Cache-Benchmark)\n1. 点查和迭代器扫描中的 Zero Copy，对大 Value 效果尤其显著\n1. 将现存的 RocksDB 组件作为**内置插件**纳入 SidePlugin 体系，例如 Cache, Comparator, TableFactory, MemTableFactory...\n1. 内置 Prometheus 指标的支持，这是在[内嵌 Http](https://github.com/topling/rockside/wiki/WebView) 中实现的\n1. 修复了很多 RocksDB 的 bug，我们已将其中易于合并到 RocksDB 的很多修复与改进给上游 RocksDB 发了 [Pull Request](https://github.com/facebook/rocksdb/pulls?q=is%3Apr+author%3Arockeet)\n\n## ToplingDB 云原生数据库服务\n1. [MyTopling](https://github.com/topling/mytopling)(MySQL on ToplingDB), [阿里云上的 MyTopling](https://market.aliyun.com/products?k=mytopling)\n1. [Todis](https://github.com/topling/todis)(Redis on ToplingDB)\n\n## ToplingDB 组件\n通过 SidePlugin 的实现机制，插件（组件）可以与 ToplingDB 的核心代码实现物理隔离\n1. 可以编译为一个单独的动态库，实现运行时动态加载\n1. 应用代码不需要为插件做任何改变，只需要修改 json/yaml 配置\n\n### git 仓库的目录结构\n```bash\ntoplingdb\n \\__ sideplugin\n      \\__ rockside                 (submodule , sideplugin core and framework)\n      \\__ topling-zip              (auto clone, zip and core lib)\n      \\__ cspp-memtab              (auto clone, sideplugin component)\n      \\__ cspp-wbwi                (auto clone, sideplugin component)\n      \\__ topling-sst              (auto clone, sideplugin component)\n      \\__ topling-rocks            (auto clone, sideplugin component)\n      \\__ topling-zip_table_reader (auto clone, sideplugin component)\n      \\__ topling-dcompact         (auto clone, sideplugin component)\n           \\_ tools/dcompact       (dcompact-worker binary app)\n```\n 仓库    | 权限 | 说明\n-------------- | ---------- | -----------\n[ToplingDB](https://github.com/topling/toplingdb) | public | 顶级仓库，分叉自 [RocksDB](https://github.com/facebook/rocksdb)，增加了我们的改进与修复\n[rockside](https://github.com/topling/rockside) | public | ToplingDB 子模块，包含：\u003cul\u003e\u003cli\u003eSidePlugin 框架和内置插件\u003c/li\u003e\u003cli\u003e内嵌的 Http 服务和 Prometheus 指标\u003c/li\u003e\u003c/ul\u003e\n[cspp-wbwi\u003cbr\u003e(**W**rite**B**atch**W**ith**I**ndex)](https://github.com/topling/cspp-wbwi) | public | 使用 Topling CSPP Trie 实现的 **CSPP_WBWI** 相比 rocksdb SkipList WBWI 最多有 20 倍以上的性能提升\n[cspp-memtable](https://github.com/topling/cspp-memtable) | public | (**CSPP** is **C**rash **S**afe **P**arallel **P**atricia trie) MemTab, 相比 SkipList：内存用量更低，单线程性能提升 7 倍，多线程线性提升，可[直接转化为 SST](https://github.com/topling/cspp-memtable#%E4%BA%8Cmemtable-%E7%9B%B4%E6%8E%A5%E8%BD%AC%E5%8C%96%E6%88%90-sst)\n[topling-sst](https://github.com/topling/topling-sst) | public | 1. [SingleFastTable](https://github.com/topling/rockside/wiki/SingleFastTable)(主要用于 L0 和 L1)\u003cbr/\u003e 2. VecAutoSortTable(主要用于 MyTopling bulk_load).\u003cbr/\u003e 3. 已弃用：[ToplingFastTable](https://github.com/topling/rockside/wiki/ToplingFastTable), CSPPAutoSortTable\n[topling-dcompact](https://github.com/topling/topling-dcompact) | public | 分布式 Compact 与通用的 dcompact_worker 程序, 将 Compact 转移到弹性计算集群。\u003cbr/\u003e相比 RocksDB 自身的 Remote Compaction，ToplingDB 的分布式 Compact 功能完备，使用便捷，对上层应用非常友好\n[topling-rocks](https://github.com/topling/topling-rocks) | **private** | 创建 [Topling**Zip**Table](https://github.com/topling/rockside/wiki/ToplingZipTable)，基于 Topling 可检索内存压缩算法的 SST，压缩率更高，且内存占用更低，一般用于 L2 及更深层 SST\n[topling-zip_table_reader](https://github.com/topling/topling-zip_table_reader) | public | 让社区版用户可以读取 Topling**Zip**Table，但创建需要私有仓库 [topling-rocks](https://github.com/topling/topling-rocks)\n\n为了简化编译流程，ToplingDB 在 Makefile 中会自动 clone 各个组件的 github 仓库，社区版用户可以成功 clone 公开的仓库，但克隆私有仓库（例如 topling-rocks）会失败，所以社区版用户编译出来的 ToplingDB 无法创建 Topling**Zip**Table，但可以读取 Topling**Zip**Table。\n\n## 可配置的功能\n为了性能和简化，ToplingDB 默认禁用了一些 RocksDB 的功能：\n\n功能|控制参数（预编译宏）\n-------|-------------\n动态创建 ColumnFamily | ROCKSDB_DYNAMIC_CREATE_CF\n用户层 timestamp | TOPLINGDB_WITH_TIMESTAMP\n宽列 | TOPLINGDB_WITH_WIDE_COLUMNS\n华而不实的功能 | TOPLINGDB_WITH_FABRICATED_COMPLEXITY\n\n为了启用这些功能，需要为 make 命令显式添加 `EXTRA_CXXFLAGS=\"-D${MACRO_1} -D${MACRO_2} ...\"`，例如编译带动态创建 ColumnFamily 的 rocksdbjava:\n```\nmake -j`nproc` EXTRA_CXXFLAGS='-DROCKSDB_DYNAMIC_CREATE_CF' rocksdbjava\n```\n\n\u003chr/\u003e\n以下是上游 RocksDB 的原版 README\n\u003chr/\u003e\n\u003chr/\u003e\n\n## RocksDB: A Persistent Key-Value Store for Flash and RAM Storage\n\n[![CircleCI Status](https://circleci.com/gh/facebook/rocksdb.svg?style=svg)](https://circleci.com/gh/facebook/rocksdb)\n[![Appveyor Build status](https://ci.appveyor.com/api/projects/status/fbgfu0so3afcno78/branch/main?svg=true)](https://ci.appveyor.com/project/Facebook/rocksdb/branch/main)\n[![PPC64le Build Status](http://140-211-168-68-openstack.osuosl.org:8080/buildStatus/icon?job=rocksdb\u0026style=plastic)](http://140-211-168-68-openstack.osuosl.org:8080/job/rocksdb)\n\nRocksDB is developed and maintained by Facebook Database Engineering Team.\nIt is built on earlier work on [LevelDB](https://github.com/google/leveldb) by Sanjay Ghemawat (sanjay@google.com)\nand Jeff Dean (jeff@google.com)\n\nThis code is a library that forms the core building block for a fast\nkey-value server, especially suited for storing data on flash drives.\nIt has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs\nbetween Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF)\nand Space-Amplification-Factor (SAF). It has multi-threaded compactions,\nmaking it especially suitable for storing multiple terabytes of data in a\nsingle database.\n\nStart with example usage here: https://github.com/facebook/rocksdb/tree/main/examples\n\nSee the [github wiki](https://github.com/facebook/rocksdb/wiki) for more explanation.\n\nThe public interface is in `include/`.  Callers should not include or\nrely on the details of any other header files in this package.  Those\ninternal APIs may be changed without warning.\n\nQuestions and discussions are welcome on the [RocksDB Developers Public](https://www.facebook.com/groups/rocksdb.dev/) Facebook group and [email list](https://groups.google.com/g/rocksdb) on Google Groups.\n\n## License\n\nRocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory).  You may select, at your option, one of the above-listed licenses.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftopling%2Ftoplingdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftopling%2Ftoplingdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftopling%2Ftoplingdb/lists"}