{"id":13411951,"url":"https://github.com/cockroachdb/pebble","last_synced_at":"2025-06-17T09:10:50.289Z","repository":{"id":37444363,"uuid":"141339868","full_name":"cockroachdb/pebble","owner":"cockroachdb","description":"RocksDB/LevelDB inspired key-value database in Go","archived":false,"fork":false,"pushed_at":"2025-06-12T17:30:45.000Z","size":73075,"stargazers_count":5337,"open_issues_count":291,"forks_count":498,"subscribers_count":101,"default_branch":"master","last_synced_at":"2025-06-12T17:34:24.383Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cockroachdb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-07-17T20:25:25.000Z","updated_at":"2025-06-12T17:23:41.000Z","dependencies_parsed_at":"2023-02-15T00:31:10.157Z","dependency_job_id":"72b5170e-d177-4f61-9ca1-f1fb6a784bd2","html_url":"https://github.com/cockroachdb/pebble","commit_stats":{"total_commits":3285,"total_committers":86,"mean_commits":38.19767441860465,"dds":0.660882800608828,"last_synced_commit":"907d86520e39c5233dbd631c98f926b4e158f050"},"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"purl":"pkg:github/cockroachdb/pebble","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cockroachdb%2Fpebble","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cockroachdb%2Fpebble/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cockroachdb%2Fpebble/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cockroachdb%2Fpebble/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cockroachdb","download_url":"https://codeload.github.com/cockroachdb/pebble/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cockroachdb%2Fpebble/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260326793,"owners_count":22992388,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T20:01:19.109Z","updated_at":"2025-06-17T09:10:50.273Z","avatar_url":"https://github.com/cockroachdb.png","language":"Go","readme":"# Pebble [![Build Status](https://github.com/cockroachdb/pebble/actions/workflows/ci.yaml/badge.svg?branch=master)](https://github.com/cockroachdb/pebble/actions/workflows/ci.yaml) [![GoDoc](https://godoc.org/github.com/cockroachdb/pebble?status.svg)](https://godoc.org/github.com/cockroachdb/pebble) \u003csup\u003e\u003csub\u003e\u003csub\u003e[Coverage](https://storage.googleapis.com/crl-codecover-public/pebble/index.html)\u003c/sub\u003e\u003c/sub\u003e\u003c/sup\u003e\n\n#### [Nightly benchmarks](https://cockroachdb.github.io/pebble/)\n\nPebble is a LevelDB/RocksDB inspired key-value store focused on\nperformance and internal usage by CockroachDB. Pebble inherits the\nRocksDB file formats and a few extensions such as range deletion\ntombstones, table-level bloom filters, and updates to the MANIFEST\nformat.\n\nPebble intentionally does not aspire to include every feature in RocksDB and\nspecifically targets the use case and feature set needed by CockroachDB:\n\n* Block-based tables\n* Checkpoints\n* Indexed batches\n* Iterator options (lower/upper bound, table filter)\n* Level-based compaction\n* Manual compaction\n* Merge operator\n* Prefix bloom filters\n* Prefix iteration\n* Range deletion tombstones\n* Reverse iteration\n* SSTable ingestion\n* Single delete\n* Snapshots\n* Table-level bloom filters\n\nRocksDB has a large number of features that are not implemented in\nPebble:\n\n* Backups\n* Column families\n* Delete files in range\n* FIFO compaction style\n* Forward iterator / tailing iterator\n* Hash table format\n* Memtable bloom filter\n* Persistent cache\n* Pin iterator key / value\n* Plain table format\n* SSTable ingest-behind\n* Sub-compactions\n* Transactions\n* Universal compaction style\n\n***WARNING***: Pebble may silently corrupt data or behave incorrectly if\nused with a RocksDB database that uses a feature Pebble doesn't\nsupport. Caveat emptor!\n\n## Production Ready\n\nPebble was introduced as an alternative storage engine to RocksDB in\nCockroachDB v20.1 (released May 2020) and was used in production\nsuccessfully at that time. Pebble was made the default storage engine\nin CockroachDB v20.2 (released Nov 2020). Pebble is being used in\nproduction by users of CockroachDB at scale and is considered stable\nand production ready.\n\n## Advantages\n\nPebble offers several improvements over RocksDB:\n\n* Faster reverse iteration via backwards links in the memtable's\n  skiplist.\n* Faster commit pipeline that achieves better concurrency.\n* Seamless merged iteration of indexed batches. The mutations in the\n  batch conceptually occupy another memtable level.\n* L0 sublevels and flush splitting for concurrent compactions out of L0 and\n  reduced read-amplification during heavy write load.\n* Faster LSM edits in LSMs with large numbers of sstables through use of a\n  copy-on-write B-tree to hold file metadata.\n* Delete-only compactions that drop whole sstables that fall within the bounds\n  of a range deletion.\n* Block-property collectors and filters that enable iterators to skip tables,\n  index blocks and data blocks that are irrelevant, according to user-defined\n  properties over key-value pairs.\n* Range keys API, allowing KV pairs defined over a range of keyspace with\n  user-defined semantics and interleaved during iteration.\n* Smaller, more approachable code base.\n\nSee the [Pebble vs RocksDB: Implementation\nDifferences](docs/rocksdb.md) doc for more details on implementation\ndifferences.\n\n## RocksDB Compatibility\n\nPebble `v1` strives for forward compatibility with RocksDB 6.2.1 (the latest\nversion of RocksDB used by CockroachDB). Forward compatibility means that a DB\ngenerated by RocksDB 6.2.1 can be upgraded for use by Pebble. Pebble versions in\nthe `v1` series may open DBs generated by RocksDB 6.2.1. Since its introduction,\nPebble has adopted various backwards-incompatible format changes that are gated\nbehind new 'format major versions'. Pebble `v2` and newer does not support\nopening DBs generated by RocksDB. DBs generated by RocksDB may only be used with\nrecent versions of Pebble after migrating them through format major version\nupgrades using previous versions of Pebble. See the below section of format\nmajor versions.\n\nEven the RocksDB-compatible versions of Pebble only provide compatibility with\nthe subset of functionality and configuration used by CockroachDB. The scope of\nRocksDB functionality and configuration is too large to adequately test and\ndocument all the incompatibilities. The list below contains known\nincompatibilities.\n\n* Pebble's use of WAL recycling is only compatible with RocksDB's\n  `kTolerateCorruptedTailRecords` WAL recovery mode. Older versions of\n  RocksDB would automatically map incompatible WAL recovery modes to\n  `kTolerateCorruptedTailRecords`. New versions of RocksDB will\n  disable WAL recycling.\n* Column families. Pebble does not support column families, nor does\n  it attempt to detect their usage when opening a DB that may contain\n  them.\n* Hash table format. Pebble does not support the hash table sstable\n  format.\n* Plain table format. Pebble does not support the plain table sstable\n  format.\n* SSTable format version 3 and 4. Pebble does not support version 3\n  and version 4 format sstables. The sstable format version is\n  controlled by the `BlockBasedTableOptions::format_version` option.\n  See [#97](https://github.com/cockroachdb/pebble/issues/97).\n\n## Format major versions\n\nOver time Pebble has introduced new physical file formats.  Backwards\nincompatible changes are made through the introduction of 'format major\nversions'. By default, when Pebble opens a database, it defaults to the lowest\nsupported version. In `v1`, this is `FormatMostCompatible`, which is\nbi-directionally compatible with RocksDB 6.2.1 (with the caveats described\nabove).\n\nDatabases created by RocksDB or Pebble versions `v1` and earlier must be upgraded\nto a compatible format major version before running newer Pebble versions. Newer\nPebble versions will refuse to open databases in no longer supported formats.\n\nTo opt into new formats, a user may set `FormatMajorVersion` on the\n[`Options`](https://pkg.go.dev/github.com/cockroachdb/pebble#Options)\nsupplied to\n[`Open`](https://pkg.go.dev/github.com/cockroachdb/pebble#Open), or\nupgrade the format major version at runtime using\n[`DB.RatchetFormatMajorVersion`](https://pkg.go.dev/github.com/cockroachdb/pebble#DB.RatchetFormatMajorVersion).\nFormat major version upgrades are permanent; There is no option to\nreturn to an earlier format.\n\nThe table below outlines the history of format major versions, along with what\nrange of Pebble versions support that format.\n\n| Name                               | Value | Migration  | Pebble support |\n|------------------------------------|-------|------------|----------------|\n| FormatMostCompatible               |   1   | No         | v1             |\n| FormatVersioned                    |   3   | No         | v1             |\n| FormatSetWithDelete                |   4   | No         | v1             |\n| FormatBlockPropertyCollector       |   5   | No         | v1             |\n| FormatSplitUserKeysMarked          |   6   | Background | v1             |\n| FormatSplitUserKeysMarkedCompacted |   7   | Blocking   | v1             |\n| FormatRangeKeys                    |   8   | No         | v1             |\n| FormatMinTableFormatPebblev1       |   9   | No         | v1             |\n| FormatPrePebblev1Marked            |  10   | Background | v1             |\n| FormatSSTableValueBlocks           |  12   | No         | v1             |\n| FormatFlushableIngest              |  13   | No         | v1, v2, master |\n| FormatPrePebblev1MarkedCompacted   |  14   | Blocking   | v1, v2, master |\n| FormatDeleteSizedAndObsolete       |  15   | No         | v1, v2, master |\n| FormatVirtualSSTables              |  16   | No         | v1, v2, master |\n| FormatSyntheticPrefixSuffix        |  17   | No         | v2, master     |\n| FormatFlushableIngestExcises       |  18   | No         | v2, master     |\n| FormatColumnarBlocks               |  19   | No         | v2, master     |\n\nUpgrading to a format major version with 'Background' in the migration\ncolumn may trigger background activity to rewrite physical file\nformats, typically through compactions. Upgrading to a format major\nversion with 'Blocking' in the migration column will block until a\nmigration is complete. The database may continue to serve reads and\nwrites if upgrading a live database through\n`RatchetFormatMajorVersion`, but the method call will not return until\nthe migration is complete.\n\nUpgrading existing stores can be performed via the `RatchetFormatMajorVersion`\nmethod. If the database does not use a custom comparer, merger, or block\nproperty collectors, the `pebble` tool can also be used, at the latest version\nthat supports the format. For example:\n```\n# WARNING: only use if no custom comparer/merger/property collector are necessary.\ngo run github.com/cockroachdb/pebble/cmd/pebble@v1.1.3 db upgrade \u003cdb-dir\u003e\n```\n\nFor reference, the table below lists the range of supported Pebble format major\nversions for CockroachDB releases.\n\n| CockroachDB release | Earliest supported                 | Latest supported            |\n|---------------------|------------------------------------|-----------------------------|\n| 20.1 through 21.1   | FormatMostCompatible               | FormatMostCompatible        |\n| 21.2                | FormatMostCompatible               | FormatSetWithDelete         |\n| 21.2                | FormatMostCompatible               | FormatSetWithDelete         |\n| 22.1                | FormatMostCompatible               | FormatSplitUserKeysMarked   |\n| 22.2                | FormatMostCompatible               | FormatPrePebblev1Marked     |\n| 23.1                | FormatSplitUserKeysMarkedCompacted | FormatFlushableIngest       |\n| 23.2                | FormatPrePebblev1Marked            | FormatVirtualSSTables       |\n| 24.1                | FormatFlushableIngest              | FormatSyntheticPrefixSuffix |\n| 24.2                | FormatVirtualSSTables              | FormatSyntheticPrefixSuffix |\n| 24.3                | FormatSyntheticPrefixSuffix        | FormatColumnarBlocks        |\n\n## Pedigree\n\nPebble is based on the incomplete Go version of LevelDB:\n\nhttps://github.com/golang/leveldb\n\nThe Go version of LevelDB is based on the C++ original:\n\nhttps://github.com/google/leveldb\n\nOptimizations and inspiration were drawn from RocksDB:\n\nhttps://github.com/facebook/rocksdb\n\n## Getting Started\n\n### Example Code\n\n```go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"log\"\n\n\t\"github.com/cockroachdb/pebble\"\n)\n\nfunc main() {\n\tdb, err := pebble.Open(\"demo\", \u0026pebble.Options{})\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tkey := []byte(\"hello\")\n\tif err := db.Set(key, []byte(\"world\"), pebble.Sync); err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tvalue, closer, err := db.Get(key)\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tfmt.Printf(\"%s %s\\n\", key, value)\n\tif err := closer.Close(); err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tif err := db.Close(); err != nil {\n\t\tlog.Fatal(err)\n\t}\n}\n```\n","funding_links":[],"categories":["Go","开源类库","Database","By Language","Key/Value Databases","Open source library","Data Integration Frameworks","数据库"],"sub_categories":["缓存","Databases Implemented in Go","Go","Cache","Go中实现的数据库"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcockroachdb%2Fpebble","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcockroachdb%2Fpebble","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcockroachdb%2Fpebble/lists"}