{"id":38869260,"url":"https://github.com/markoelez/basicfs","last_synced_at":"2026-01-17T14:28:13.207Z","repository":{"id":126785290,"uuid":"289055185","full_name":"markoelez/basicfs","owner":"markoelez","description":"Simple distributed key-value store for small files","archived":false,"fork":false,"pushed_at":"2024-02-01T00:05:25.000Z","size":57,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-02-02T00:33:40.011Z","etag":null,"topics":["distributed-storage","distributed-systems"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/markoelez.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-08-20T16:27:24.000Z","updated_at":"2024-02-02T00:33:40.012Z","dependencies_parsed_at":"2023-06-18T01:45:18.386Z","dependency_job_id":null,"html_url":"https://github.com/markoelez/basicfs","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/markoelez/basicfs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/markoelez%2Fbasicfs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/markoelez%2Fbasicfs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/markoelez%2Fbasicfs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/markoelez%2Fbasicfs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/markoelez","download_url":"https://codeload.github.com/markoelez/basicfs/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/markoelez%2Fbasicfs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28509948,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T13:38:16.342Z","status":"ssl_error","status_checked_at":"2026-01-17T13:37:44.060Z","response_time":85,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-storage","distributed-systems"],"created_at":"2026-01-17T14:28:12.489Z","updated_at":"2026-01-17T14:28:13.201Z","avatar_url":"https://github.com/markoelez.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# BasicFS\n\n[![CI](https://github.com/markoelez/basicfs/actions/workflows/ci.yaml/badge.svg)](https://github.com/markoelez/basicfs/actions/workflows/ci.yaml)\n\n\n## Introduction\n\nBasicFS is a very simple distributed key value store optimized for small files (i.e. photos), inspired by Facebook's [Haystack](https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf) object store and [SeaweedFS](https://github.com/chrislusf/seaweedfs).\n\n## Usage\n\nBy default, volume servers will run on port 9091. When using multiple volume servers, their respective ports should be specified. The master server will default to port 9090 and should be initialized with a comma separated string containing all volume server urls.\n\n### Build and run docker image\n\n```\n./docker.sh\n```\nDependencies are tracked/installed using [Pipenv](https://pipenv.pypa.io/en/stable/) and [Pipfile](https://github.com/pypa/pipfile).\n\nDependencies can be installed manually using:\n\n```\npipenv lock --requirements \u003e requirements.txt \u0026\u0026 pip install -r /tmp/requirements.txt\n```\n\n### Start Two Volume Servers\n\n```\nPORT=9090 VOLUME=/tmp/v1 ./scripts/volume\nPORT=9091 VOLUME=/tmp/v1 ./scripts/volume\n```\n\n### Start Master Server\n\nMust have more volumes than replicas. \n\n```\nPORT=9092 DB=/tmp/db REPLICAS=2 ./scripts/master localhost:9090,localhost:9091\n```\n\n### Write File\n\nTo write a file, send a HTTP PUT request containing the filedata to the master server.\n\n```\ncurl -X PUT -d filedata localhost:9092/file_id\n```\n\n### Read File\n\nTo read a file, send a HTTP GET request to the `file_id`.\n\n```\ncurl localhost:9092/file_id\n```\n\nYou can use also use this URL to read directly from the volume server:\n\n```\nhttp://localhost:9090/file_id\n```\n\n### Delete File\n\nTo delete a file, send a HTTP DELETE request to the `file_id`.\n\n```\ncurl -X DELETE localhost:9092/file_id\n```\n\n## Architecture\n\nBasicFS is designed to handle small files efficiently.\n\nCurrently file_id's are mapped to volume servers with the master. Eventually, this should be changed so that the master is only aware of volumeIDs which are mapped to their respective urls. Since objects will be written once and read often, the file_id and volumeID mapping should be cached in a local database after the initial write and used in subsequent GET requests. Uploaded key/value pairs will be replicated accross the specified volume servers based on the user specified replication protocol.\n\n## Todo\n- Consistent hashing instead of random volume selection\n- RAFT consensus protocol\n- RPC communication for master --\u003e volume relationship (using gRPC, protocol buffers)\n- Allow for incorporation of additional volumes to master index (using rebuild, RPC heartbeat)\n\n## License\n\nAll code is MIT licensed. Libraries follow their respective licenses.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarkoelez%2Fbasicfs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarkoelez%2Fbasicfs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarkoelez%2Fbasicfs/lists"}