{"id":17679892,"url":"https://github.com/helyim/helyim","last_synced_at":"2025-10-03T14:20:42.125Z","repository":{"id":189927015,"uuid":"680718897","full_name":"helyim/helyim","owner":"helyim","description":"seaweedfs implemented in pure Rust","archived":false,"fork":false,"pushed_at":"2025-04-01T02:09:05.000Z","size":386,"stargazers_count":188,"open_issues_count":15,"forks_count":20,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-05-20T08:04:28.585Z","etag":null,"topics":["cuda","dpdk","erasure-coding","hdfs","iouring","kernel-bypass","object-storage","rdma","s3","spdk","webdav"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/helyim.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-20T06:59:28.000Z","updated_at":"2025-05-09T08:58:12.000Z","dependencies_parsed_at":"2023-12-21T12:34:37.892Z","dependency_job_id":"a76a5b9c-e465-4b1d-8aac-ce002775d05e","html_url":"https://github.com/helyim/helyim","commit_stats":null,"previous_names":["helyim/helyim"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/helyim/helyim","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/helyim%2Fhelyim","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/helyim%2Fhelyim/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/helyim%2Fhelyim/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/helyim%2Fhelyim/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/helyim","download_url":"https://codeload.github.com/helyim/helyim/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/helyim%2Fhelyim/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278173725,"owners_count":25942363,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-03T02:00:06.070Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cuda","dpdk","erasure-coding","hdfs","iouring","kernel-bypass","object-storage","rdma","s3","spdk","webdav"],"created_at":"2024-10-24T09:04:35.883Z","updated_at":"2025-10-03T14:20:42.095Z","avatar_url":"https://github.com/helyim.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n    \u003ch1\u003eHelyim\u003c/h1\u003e\n    \u003ch4\u003e\n        \u003ca href=\"https://github.com/seaweedfs/seaweedfs\"\u003eseaweedfs\u003c/a\u003e implemented in pure Rust. Please ⭐ on \u003ca href=\"https://github.com/helyim/helyim\"\u003egithub\u003c/a\u003e!\n    \u003c/h4\u003e\n\n[![Crates.io](https://img.shields.io/crates/v/helyim.svg)](https://crates.io/crates/helyim)\n[![CI](https://github.com/helyim/helyim/actions/workflows/build.yaml/badge.svg)](https://github.com/helyim/helyim/actions/workflows/build.yaml)\n[![codecov](https://codecov.io/github/helyim/helyim/graph/badge.svg?token=QRH8H0O6P5)](https://codecov.io/github/helyim/helyim)\n![License](https://img.shields.io/badge/license-MIT%2FApache--2.0-blue)\n\n\u003c/div\u003e\n\n### Features\n\n#### Additional Features\n\n- [x] Can choose no replication or different replication levels, rack and data center aware.\n- [x] Automatic compaction to reclaim disk space after deletion or update.\n- [x] Automatic master servers fail over - no single point of failure (SPOF).\n- [x] **Erasure Coding** for warm storage Rack-Aware 10.4 erasure coding reduces storage cost.\n\n### Installation\n\n```shell\ncargo +nightly install helyim\n```\n\n### Usage\n\nBy default, the master node runs on port 9333, and the volume nodes run on port 8080. Let's start one master node, and one volume node on port 8080. Ideally, they should be started from different machines. We'll use localhost as an example.\n\nHelyim uses HTTP REST operations to read, write, and delete. The responses are in JSON or JSONP format.\n\n#### 1. Start Master Server\n\n```shell\ncargo run --release --bin helyim master\n```\n\n#### 2. Start Volume Servers\n\n```shell\ncargo run --release --bin helyim volume --port 8080 --folders ./target\n```\n\n#### 3. Write File\n\nTo upload a file: first, send a HTTP POST, PUT, or GET request to `/dir/assign` to get an `fid` and a volume server URL:\n\n```bash\ncurl http://localhost:9333/dir/assign\n{\"fid\":\"6,16b7578a5\",\"url\":\"127.0.0.1:8080\",\"public_url\":\"127.0.0.1:8080\",\"count\":1,\"error\":\"\"}\n```\n\nSecond, to store the file content, send a HTTP multipart POST request to `url + '/' + fid` from the response:\n\n```bash\ncurl -F file=@./sun.jpg http://127.0.0.1:8080/6,16b7578a5\n{\"name\":\"sun.jpg\",\"size\":1675569,\"error\":\"\"}\n```\n\nTo update, send another POST request with updated file content.\n\nFor deletion, send an HTTP DELETE request to the same `url + '/' + fid` URL:\n\n```bash\ncurl -X DELETE http://127.0.0.1:8080/6,16b7578a5\n```\n\n#### 4. Start Filer Server\n\n```shell\ncargo run --release --bin helyim filer\n```\n\nTo upload a file without assigning file id\n\n```shell\ncurl -F file=@./sun.jpg http://127.0.0.1:8888/sun.jpg\n```\n\n### Failover Master Server\n\nWhen initiating a Raft cluster, it is necessary to specify the same node sequence when starting the Leader and Follower instances.\n\nYou can view the cluster status by accessing `http://localhost:9333/cluster/status`.\n\n```bash\n# start master1\ncargo run --bin helyim -- master --ip 127.0.0.1 --port 9333 \\\n      --peers 127.0.0.1:9333 \\\n      --peers 127.0.0.1:9335 \\\n      --peers 127.0.0.1:9337\n      \n      \n# start master2\ncargo run --bin helyim -- master --ip 127.0.0.1 --port 9335 \\\n      --peers 127.0.0.1:9333 \\\n      --peers 127.0.0.1:9335 \\\n      --peers 127.0.0.1:9337\n      \n# start master3\ncargo run --bin helyim -- master --ip 127.0.0.1 --port 9337 \\\n      --peers 127.0.0.1:9333 \\\n      --peers 127.0.0.1:9335 \\\n      --peers 127.0.0.1:9337\n```\n\n### Benchmark\n\nMy laptop results on Lenovo IdeaPad Pro 16 (2023) with SSD, CPU: 14 Intel Core i9 5.4GHz.\n\nIt seems to be slower than `seaweedfs`, especially in terms of reading.\n\n```console\n➜ ./weed benchmark -server=localhost:9333\nThis is SeaweedFS version 0.76 linux amd64\n\n------------ Writing Benchmark ----------\nCompleted 15199 of 1048576 requests, 1.4% 15198.1/s 15.3MB/s\nCompleted 31887 of 1048576 requests, 3.0% 16687.9/s 16.8MB/s\nCompleted 48439 of 1048576 requests, 4.6% 16551.6/s 16.7MB/s\n...\nCompleted 994044 of 1048576 requests, 94.8% 16645.2/s 16.8MB/s\nCompleted 1010800 of 1048576 requests, 96.4% 16755.8/s 16.9MB/s\nCompleted 1027412 of 1048576 requests, 98.0% 16612.2/s 16.7MB/s\nCompleted 1044319 of 1048576 requests, 99.6% 16907.0/s 17.0MB/s\n\nConcurrency Level:      16\nTime taken for tests:   63.249 seconds\nComplete requests:      1048576\nFailed requests:        0\nTotal transferred:      1106759553 bytes\nRequests per second:    16578.50 [#/sec]\nTransfer rate:          17088.29 [Kbytes/sec]\n\nConnection Times (ms)\n              min      avg        max      std\nTotal:        0.1      0.9       29.8      0.4\n\nPercentage of the requests served within a certain time (ms)\n   50%      0.9 ms\n   66%      1.0 ms\n   75%      1.1 ms\n   90%      1.3 ms\n   95%      1.5 ms\n   98%      1.7 ms\n   99%      1.8 ms\n  100%     29.8 ms\n\n------------ Randomly Reading Benchmark ----------\nCompleted 89963 of 1048576 requests, 8.6% 89957.6/s 90.5MB/s\nCompleted 187560 of 1048576 requests, 17.9% 97597.1/s 98.2MB/s\nCompleted 283486 of 1048576 requests, 27.0% 95925.8/s 96.6MB/s\nCompleted 382035 of 1048576 requests, 36.4% 98549.4/s 99.2MB/s\nCompleted 480649 of 1048576 requests, 45.8% 98613.9/s 99.3MB/s\nCompleted 583585 of 1048576 requests, 55.7% 102933.7/s 103.6MB/s\nCompleted 683954 of 1048576 requests, 65.2% 100370.9/s 101.0MB/s\nCompleted 782522 of 1048576 requests, 74.6% 98567.9/s 99.2MB/s\nCompleted 883504 of 1048576 requests, 84.3% 100982.7/s 101.7MB/s\nCompleted 987320 of 1048576 requests, 94.2% 103814.3/s 104.5MB/s\n\nConcurrency Level:      16\nTime taken for tests:   10.600 seconds\nComplete requests:      1048576\nFailed requests:        0\nTotal transferred:      1106777459 bytes\nRequests per second:    98925.73 [#/sec]\nTransfer rate:          101969.36 [Kbytes/sec]\n\nConnection Times (ms)\n              min      avg        max      std\nTotal:        0.0      0.1       2.3      0.1\n\nPercentage of the requests served within a certain time (ms)\n   50%      0.1 ms\n   95%      0.2 ms\n   98%      0.4 ms\n  100%      2.3 ms\n```\n\n### Acknowledgments\n\n- [seaweedfs](https://github.com/seaweedfs/seaweedfs) - SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.\n- [zergling](https://github.com/july2993/zergling) - seaweedFS re-implemented in Rust.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhelyim%2Fhelyim","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhelyim%2Fhelyim","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhelyim%2Fhelyim/lists"}