{"id":17082017,"url":"https://github.com/studyzy/shardingdb","last_synced_at":"2025-09-18T22:53:26.386Z","repository":{"id":194944728,"uuid":"691853647","full_name":"studyzy/shardingdb","owner":"studyzy","description":"LevelDB support sharding and concurrent writes","archived":false,"fork":false,"pushed_at":"2023-10-30T09:37:13.000Z","size":147,"stargazers_count":9,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-12T21:06:45.674Z","etag":null,"topics":["golang","goleveldb","leveldb","sharding","sharding-db"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/studyzy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-09-15T03:10:04.000Z","updated_at":"2025-01-26T08:16:08.000Z","dependencies_parsed_at":null,"dependency_job_id":"8f8c8c63-c603-4f27-add1-030c3b744f95","html_url":"https://github.com/studyzy/shardingdb","commit_stats":null,"previous_names":["studyzy/goleveldb-sharding"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/studyzy%2Fshardingdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/studyzy%2Fshardingdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/studyzy%2Fshardingdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/studyzy%2Fshardingdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/studyzy","download_url":"https://codeload.github.com/studyzy/shardingdb/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248631683,"owners_count":21136562,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","goleveldb","leveldb","sharding","sharding-db"],"created_at":"2024-10-14T12:58:46.166Z","updated_at":"2025-09-18T22:53:21.329Z","avatar_url":"https://github.com/studyzy.png","language":"Go","readme":"# shardingdb\n\nShardingDB is an open-source, sharded database enhancing LevelDB with concurrent reads/writes support. \nIt significantly improves performance, boosting PutData by 60x and GetData by 7x, making it an ideal drop-in replacement for LevelDB.\n-----------\n\n## Requirements\n\n* Requires at least `go1.14` or newer.\n\n## How to use\n\n### 1. Resharding\n#### 1.0 Build the resharding tool\n```bash\nmake\ncd bin\n```\n#### 1.1 Migrate data from LevelDB to new shardingdb\nFor example, if you have 1 LevelDB data and want to migrate it to 3 shardingdb data, print summary log(1), you can run the following command:\n```bash\n./resharding -i /data1 -o /newfolder1,/newfolder2,/newfolder3 -l 1\n```\n#### 1.2 Add sharding db\nFor example, if you have 1 LevelDB data and want to add 2 more LevelDB folders to shardingdb, print no log(0), you can run the following command:\n```bash\n\n```bash\n./resharding -i /data1 -o /data1,/data2,/data3 \n```\n\nFor example, if you have 3 LevelDB data and want to add 1 more LevelDB folder to shardingdb, print detail log(2), you can run the following command:\n```bash\n\n```bash\n./resharding -i /data1,/data2,/data3 -o /data1,/data2,/data3,/data4 -l 2\n```\n\n### 2. Code example\n#### 2.0 Get the package\n```go\ngo get github.com/studyzy/shardingdb\n```\n#### 2.1 Import the package\n```go\nimport \"github.com/studyzy/shardingdb\"\n```\n#### 2.2 Use shardingdb\n```go\ninputPathList := []string{\"/data1\", \"/data2\"}\nsdb, err := shardingdb.OpenFile(inputPathList, nil)\nsdb.Put([]byte(\"key\"), []byte(\"value\"), nil)\nsdb.Get([]byte(\"key\"), nil)\n...\n```\n### 2.3 Another example\n```go\ndb1, err := leveldb.OpenFile(getTempDir(), nil)\nif err != nil {\n    t.Fatal(err)\n}\ndb2, err := leveldb.OpenFile(getTempDir(), nil)\nif err != nil {\n    t.Fatal(err)\n}\n// Create a new sharding db\nsdb, err := shardingdb.NewShardingDb(shardingdb.WithDbHandles(db1,db2), shardingdb.WithShardingFunc(MurmurSharding))\n...\n```\n\n## Performance Benchmark\n\n### Environment\n* Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz  * 10 Core\n* 40GB RAM\n* 3 SSD: /data, /data1, /\n\ngenerate data by command:\n```bash\ngo test -timeout 60m -run \"TestCompareDbPerformance\"\n```\nTest case: total 1000000 key-value pairs, 100 go routines, 100 key-value pairs per batch.\nresult means the time cost(second) of the whole operation.\n### 1. PutData\n\n| Data Size | LevelDB | ShardingDB(3 folders) | ShardingDB(6 folders) | ShardingDB(encrypt 3 folders) |\n|:---------:|:-------:|:---------------------:|:---------------------:|:-----------------------------:|\n|   100B    |  2.27  |        0.659         |        0.581         |            0.953             |\n|   200B    |  4.45  |         1.07         |        0.683         |             1.9              |\n|   500B    |  15.3  |         3.36         |         1.49         |             6.4              |\n|    1KB    |  48.9  |         9.42         |         3.74         |            17.69             |\n|   10KB    |  1117  |         351          |         123          |             308              |\n\n### 2. GetData\n\n| Data Size | LevelDB | ShardingDB(3 folders) | ShardingDB(6 folders) | ShardingDB(encrypt 3 folders) |\n|:---------:|:-------:|:---------------------:|:---------------------:|:-----------------------------:|\n|   100B    |  2.23  |         1.25         |         1.02         |             1.86             |\n|   200B    |  3.09  |         1.42         |         1.27         |             2.24             |\n|   500B    |  4.17  |         1.91         |         1.62         |             3.73             |\n|    1KB    |  7.97  |         2.37         |         2.26         |             4.53             |\n|   10KB    | 12.75  |         9.54         |        11.03         |            13.85             |\n\n### 3. GetData not found\n\n| Data Size | LevelDB | ShardingDB(3 folders) | ShardingDB(6 folders) | ShardingDB(encrypt 3 folders) |\n|:---------:|:-------:|:---------------------:|:---------------------:|:-----------------------------:|\n|   100B    |  2.14  |         1.36         |         0.87         |             1.43             |\n|   200B    |  2.07  |         1.47         |         0.9          |             1.6              |\n|   500B    |  2.05  |         1.51         |         0.93         |             1.81             |\n|    1KB    |  2.35  |         1.64         |        0.891         |             2.28             |\n|   10KB    |  8.68  |         5.56         |         2.48         |             7.75             |\n\n\n### 4. DeleteData\n\n| Data Size | LevelDB | ShardingDB(3 folders) | ShardingDB(6 folders) | ShardingDB(encrypt 3 folders) |\n|:---------:|:-------:|:---------------------:|:---------------------:|:-----------------------------:|\n|   100B    |  3.82  |         2.76         |         1.02         |             1.72             |\n|   200B    |  3.81  |         1.71         |         1.02         |             1.74             |\n|   500B    |  3.85  |         1.76         |         1.05         |             1.69             |\n|    1KB    |  3.84  |         1.72         |         1.04         |             1.74             |\n|   10KB    | 3.844  |         1.78         |         1.06         |             1.76             |\n\n### 5. Iterator\n\n| Data Size | LevelDB | ShardingDB(3 folders) | ShardingDB(6 folders) | ShardingDB(encrypt 3 folders) |\n|:---------:|:-------:|:---------------------:|:---------------------:|:-----------------------------:|\n|   100B    | 0.133  |        0.184         |        0.222         |             0.18             |\n|   200B    | 0.151  |        0.246         |        0.246         |            0.191             |\n|   500B    | 0.282  |        0.351         |         0.41         |            0.344             |\n|    1KB    | 0.514  |        0.419         |        0.472         |            0.541             |\n|   10KB    |  2.46  |         2.39         |         1.96         |             2.3              |\n\n\n### 6. Sharding count compare\nrun command:\n```bash\ngo test  -timeout 60m -run \"TestCompareShardingCountPerformance\"\n```\n\n#### 6.1 PutData\n| Data Size | ShardingDB(3 folders) | ShardingDB(6 folders) | ShardingDB(9 folders) | ShardingDB(30 folders) | ShardingDB(60 folders) |\n|:---------:|:---------------------:|:---------------------:|:---------------------:|:----------------------:|:----------------------:|\n|   100B    |        0.659         |        0.581         |        0.506         |         0.564         |         0.728         |\n|   200B    |         1.07         |        0.683         |        0.624         |         0.685         |         0.782         |\n|   500B    |         3.36         |         1.49         |         1.20         |         1.18          |         1.21          |\n|    1KB    |         9.42         |         3.74         |         2.33         |         1.92          |         1.96          |\n|   10KB    |         351          |         123          |          54          |          26           |         18.2          |\n#### 6.2 GetData\n| Data Size | ShardingDB(3 folders) | ShardingDB(6 folders) | ShardingDB(9 folders) | ShardingDB(30 folders) | ShardingDB(60 folders) |\n|:---------:|:---------------------:|:---------------------:|:---------------------:|:----------------------:|:----------------------:|\n|   100B    |         1.25         |         1.02         |         1.03         |         0.343         |         0.366         |\n|   200B    |         1.42         |         1.27         |         1.01         |         0.66          |         0.373         |\n|   500B    |         1.91         |         1.62         |         1.21         |         0.96          |         1.34          |\n|    1KB    |         2.37         |         2.26         |         1.83         |         1.18          |         1.19          |\n|   10KB    |         9.54         |        11.03         |         7.67         |          4.8          |          3.4          |\n\n\nMost interfaces are the same as [goleveldb](https://github.com/syndtr/goleveldb). For my interface definition, please refer to [DbHandle](https://github.com/studyzy/shardingdb/blob/main/interfaces.go).","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstudyzy%2Fshardingdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstudyzy%2Fshardingdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstudyzy%2Fshardingdb/lists"}