{"id":19347247,"url":"https://github.com/tinoryj/cdstore","last_synced_at":"2025-02-24T09:43:55.793Z","repository":{"id":115603588,"uuid":"126451408","full_name":"tinoryj/CDStore","owner":"tinoryj","description":"System prototype for USENIX ATC 2015: \"Convergent Dispersal Deduplication Datastore\"","archived":false,"fork":false,"pushed_at":"2019-04-01T11:41:51.000Z","size":25244,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-06T14:35:26.738Z","etag":null,"topics":["deduplication","leveldb"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tinoryj.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-03-23T07:54:01.000Z","updated_at":"2021-04-12T05:44:27.000Z","dependencies_parsed_at":null,"dependency_job_id":"ad08ef58-cd3c-4917-bd43-ad1b7d3fd887","html_url":"https://github.com/tinoryj/CDStore","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinoryj%2FCDStore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinoryj%2FCDStore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinoryj%2FCDStore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinoryj%2FCDStore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tinoryj","download_url":"https://codeload.github.com/tinoryj/CDStore/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240457940,"owners_count":19804489,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deduplication","leveldb"],"created_at":"2024-11-10T04:15:10.747Z","updated_at":"2025-02-24T09:43:55.750Z","avatar_url":"https://github.com/tinoryj.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CDStore: Toward Reliable, Secure, and Cost-Efficient Cloud Storage via Convergent Dispersal\n   \n## Introduction\nCDStore builds on an augmented secret sharing scheme called convergent dispersal, which supports deduplication by using deterministic content-derived hashes as inputs to secret sharing. It combines convergent dispersal with two-stage deduplication to achieve both bandwidth and storage savings and be robust against side-channel attacks. \n\n### Publications\n- Mingqiang Li, Chuan Qin, and Patrick P. C. Lee \n\t**\"CDStore: Toward Reliable, Secure, and Cost-Efficient Cloud Storage via Convergent Dispersal.\"** \n\tProceedings of USENIX Annual Technical Conference (ATC 2015), Santa Clara, CA, July 2015.\n\n- Mingqiang Li, Chuan Qin, Jingwei Li, and Patrick P. C. Lee.\n\t**\"CDStore: Toward Reliable, Secure, and Cost-Efficient Cloud Storage via Convergent Dispersal.\"**\n\tIEEE Internet Computing, 20(3), pp. 45-53, May-June 2016 (Special issue: Cloud Storage).\n\t(An earlier version appeared in USENIX ATC 2015)\n\n## Dependencies\nCDStore is built on Ubuntu 12.04.3 LTS with gcc version 4.6.3. This software requires the following libraries:\n\n * OpenSSL (https://www.openssl.org/source/openssl-1.0.2a.tar.gz)\n * GF-Complete (https://github.com/ceph/gf-complete/archive/master.zip)\n * boost C++ library (http://sourceforge.net/projects/boost/files/boost/1.58.0/boost_1_58_0.tar.gz)\n * Leveldb (https://github.com/google/leveldb/archive/master.zip)\n\nGF-Complete and Leveldb have been packed in `client/lib/` and `server/lib/` respectively.\n\n## Installation\n\nInstall OpenSSL, Boost C++ library and snappy (that is necessary for Leveldb) by the following command:\n\n```\n$ sudo apt-get install libssl1.0.0 libboost-all-dev libsnappy-dev\n```\n\nType the following commands to compile GF_Complete in `client/lib/gf_complete`:\n\n```\n$ cd client/lib/gf_complete\n$ ./configure\n$ make \n$ sudo make install\n```\n\n## Configurations\n\nAfter installing the necessary libraries, following the instructions to configure CDStore server and client.  \n\n### Server\n\nCDStore requires at least 4 servers for reliability. You need to configure and `make` them, respectively. In a successfully configured server, it has a directory `server/meta/` which stores the deduplication index, file recipes and share containers. \n\nAfter a successful `make`, you will get the server program `SERVER`. You can start it by the following command, where `[port]` is the port that CDStore server serves in.  \n\n```\n$ ./SERVER [port]\n```\n\n### Client\n\nModify the configuration file `client/config` to specify the server information. For example, if you have run 4 servers with `./SERVER [port]` on machines: \n\n```\n0.0.0.0 with port 11030\n0.0.0.0 with port 11031\n0.0.0.0 with port 11032\n0.0.0.0 with port 11033\n```\n\nYou need to change `client/config` to:\n\n```\n0.0.0.0:11030\n0.0.0.0:11031\n0.0.0.0:11032\n0.0.0.0:11033\n```\n\nOptionally, you can make advanced configure by modifying the total number of servers (4 by default), fault tolerance degree (1 by default), and security degree (0 by default) in `client/config`.\n\nAfter all configurations, change directory to `client/` and type `make` to create client program `CLIENT`.  \n\n### Quick Start\n\nWe provide scripts `auto_config.sh` and `auto_clean.sh` for quick configurations. You can run `./auto_config.sh` to compile and configure 4 servers, and run `./auto_clean.sh` to reset all configurations. \n\n## Usage\n\nYou can use the executable file `CLIENT` (in `client/`) in the following way:\n\n```\nusage: ./CLIENT [filename] [userID] [action] [secutiyType]\n\n- [filename]: full path of the file;\n- [userID]: user ID of current client;\n- [action]: [-u] upload; [-d] download;\n- [securityType]: [HIGH] AES-256 \u0026 SHA-256; [LOW] AES-128 \u0026 SHA-1\n```\n\nWe show two usage examples:  \n\n```\n// upload a file `test` from user 0 using high security mechanism (e.g., AES-256 \u0026 SHA-256)\n$ ./CLIENT test 0 -u HIGH\n\n// download a file `test`, from user 1 using low security mechanism (e.g., AES-128 \u0026 SHA-1)\n// the downloaded file will be renamed to be test.d automatically\n$ ./CLIENT test 1 -d LOW\n```\n\n## Limitations \u0026 Known Bugs\n\n- We assume the upload and download channels are secure (e.g., encrypted and authenticated), and do not implement mechanism for protection. We also assume that the user ID (input by CDStore client) is correct, so as to ensure the two-stage deduplication robust against side-channel attacks.    \n\n- When encode a test file generated via `urandom`, CDStore sometimes aborts with some errors like \n\t```\n\tError in region multiply operation.\n\tThe source \u0026 destination pointers must be aligned with respect to each other along a 16 byte boundary.\n\t``` \n\tThe error is possibly due to the library `gf_complete` that cannot encode random content.\n\n\n\n## Maintainers\n * Current maintainer\n\t- Yanjing Ren, UESTC, tinoryj@gmail.com\n\t- JinGang Ma, UESTC, demon64523@gmail.com\n\n * Original maintainer\n\t- Chuan Qin, CUHK, chintran27@gmail.com\n\t- Mingqiang Li, CUHK, mingqianglicn@gmail.com\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftinoryj%2Fcdstore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftinoryj%2Fcdstore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftinoryj%2Fcdstore/lists"}