{"id":21597273,"url":"https://github.com/spheresoftware/objstore","last_synced_at":"2026-03-09T12:38:26.989Z","repository":{"id":57613967,"uuid":"101626563","full_name":"SphereSoftware/objstore","owner":"SphereSoftware","description":"A Multi-Master Distributed Caching Layer for Amazon S3.","archived":false,"fork":false,"pushed_at":"2020-10-22T06:39:55.000Z","size":724,"stargazers_count":74,"open_issues_count":0,"forks_count":4,"subscribers_count":165,"default_branch":"master","last_synced_at":"2025-04-11T00:52:09.657Z","etag":null,"topics":["amazon","astranet","cache-storage","distributed-systems","go","golang","s3"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SphereSoftware.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-08-28T09:22:00.000Z","updated_at":"2024-11-11T02:49:51.000Z","dependencies_parsed_at":"2022-09-11T01:00:45.055Z","dependency_job_id":null,"html_url":"https://github.com/SphereSoftware/objstore","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SphereSoftware%2Fobjstore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SphereSoftware%2Fobjstore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SphereSoftware%2Fobjstore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SphereSoftware%2Fobjstore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SphereSoftware","download_url":"https://codeload.github.com/SphereSoftware/objstore/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248322609,"owners_count":21084336,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon","astranet","cache-storage","distributed-systems","go","golang","s3"],"created_at":"2024-11-24T18:08:36.914Z","updated_at":"2026-03-09T12:38:26.960Z","avatar_url":"https://github.com/SphereSoftware.png","language":"Go","readme":"\u003cp align=\"center\"\u003e\n    \u003cimg align=\"center\" src=\"docs/logo.png\" height=\"200px\" alt=\"objstore near cache layer\" /\u003e\n\u003c/p\u003e\n\n# objstore [![Go Report Card](https://goreportcard.com/badge/sphere.software/objstore)](https://goreportcard.com/report/sphere.software/objstore) ![Status Badge](https://img.shields.io/badge/status-open--beta-orange.svg)\n_A Multi-Master Distributed Caching Layer for Amazon S3_\n\nThis project aims to provide an easy to use, self-organising multi-master caching layer for various cloud stoarge backends, e.g. S3. It combines functionality of a simple object storage with added robustness of cross-node journal synchronisation, object replication and cluster auto-discovery.\n\nWe know that Amazon S3 has proven to be fast and reliable, a PaaS solution that acts like a\nbackbone for many business applications. But the cost of service may become too high\ndepending on your usage patterns, for example, if your application runs in your own datacenter, then\nthe file transfer costs will skyrocket. Also request frequency has its limits.\n\n**Objstore Cluster** aims to mitigate this problem, it's supposed to be running in your datacenter, implementing a near-cache for all files. Its API allows to upload, head, read and delete files by key, like any other object. All related meta-data may be perserved with files as well. This caching layer will upload the file to S3 and store a copy locally, with optional replication among other nodes. Next time you'd access the file, it will be served from a local machine, or its near nodes, in case of a cache miss, it will get the file from S3 directly.\n\n\u003cp align=\"center\"\u003e\n    \u003cimg align=\"center\" src=\"docs/cluster-view-0.png\" width=\"600px\" alt=\"objstore cluster overview\" /\u003e\n\u003c/p\u003e\n\nThe cluster must be robust, altrough it's not required to reach the same levels as traditional DBs or other stores that are required to be highly consistent, a certant amount of fault resilience is important because a dropped cache implies a huge (and unplanned) spike in latency and CoS, which may hurt infrastructure and your wallet. And caches may recover very slowly.\n\nObjstore leverages a P2P discovery mechanism, so once some nodes are started already, another one might join knowing only one physical IP address. The cluster setups a logical network over persistent TCP connections between nodes and uses an internal HTTP API to share events and data between nodes, eliminating the single point of failure. Everything involves zero configuration, except the HTTP load balancer which may be any of your choice.\n\n\u003cp align=\"center\"\u003e\n    \u003cimg align=\"center\" src=\"docs/cluster-view-1.png\" width=\"600px\" alt=\"objstore cluster zoom\" /\u003e\n\u003c/p\u003e\n\nNode disk sizes are required to be identical, the overall limit of the cluster is limited by size of the smallest disk used for data replication. If you want to expand the size linearly, setup another Object Store cluster and tweak your HTTP load balancer.\n\n## Installation\n\n```\ngo get -u sphere.software/objstore/cmd/objstore\n```\n\nFor local Docker builds:\n\n```\ncd deploy \u0026\u0026 make local\n```\n\nFor remote / CI Docker builds under Linux:\n\n```\ncd deploy \u0026\u0026 make build\n```\n\n## Server usage\n\n```\n$ objstore -h\n\nUsage: objstore [OPTIONS]\n\nA Multi-Master Distributed Caching Layer for Amazon S3.\nVersion 0.1 http://github.com/SphereSoftware/objstore\n\nOptions:\n  -d, --debug                       Debug level to use, currently 0/1 suppported. ($APP_DEBUG_LEVEL)\n  -N, --nodes                       A list of cluster nodes to join for discovery and journal updates ($APP_CLUSTER_NODES)\n  -T, --tag=\"default\"               Cluster tag name ($APP_CLUSTER_TAGNAME)\n  --private-addr=\"0.0.0.0:11999\"    Listen address for cluster discovery and private API ($NET_PRIVATE_ADDR)\n  --debug-addr=\"\"                   Listen address for private API debugging using external tools ($NET_DEBUG_ADDR)\n  --public-addr=\"0.0.0.0:10999\"     Listen address for external access and public HTTP API ($NET_PUBLIC_ADDR)\n  --state-dir=\"state/\"              Directory where to keep local state and journals. ($APP_STATE_DIR)\n  --files-dir=\"files/\"              Directory where to keep local files. ($APP_FILES_DIR)\n  -R, --region=\"us-east-1\"          Amazon S3 region name ($S3_REGION_NAME)\n  -B, --bucket=\"00-objstore-test\"   Amazon S3 bucket name ($S3_BUCKET_NAME)\n```\n\nExample use, single node:\n\n```bash\n$ objstore -d 1 # with debug\n\n[INFO] node ID: 01BRNEKEZGKFSPAT10KZM5A141\n[WARN] no additional cluster nodes specified, current node starts solo\n[GIN-debug] Listening and serving HTTP on 0.0.0.0:10999\n```\n\nYou can start another nodes, on the same machine or another. If starting on the same machine, make sure that data directories are not colliding and the private/public API ports are different. To start a node that will join to the cluster with first one:\n\n```bash\n$ objstore -d 1 -N localhost:11999 \\\n    --private-addr=\"0.0.0.0:11997\" --public-addr=\"0.0.0.0:10997\"\n\n[INFO] node ID: 01BRNKZ01MFSJJDN98F6M0640K\n[GIN-debug] Listening and serving HTTP on 0.0.0.0:10997\n[INFO] joining to cluster [localhost:11999]\n[INFO] sync done\n```\n\nBy checking both nodes logs, you can see that `/private/v1/sync` has been called from each other. After that journals are in sync. More about journal synchronisation and node failure scenarios will be written soon in a standalone document.\n\n## Client usage\n\nAt this moment both nodes are listening on the public HTTP API addresses:\n\n* `localhost:10999`\n* `localhost:10997`\n\nYou don't need to use a load balancer to start utilising the cluster, the requests may be directed to any active node in the cluster. Load balancer would allow to split the workload equally, also it helps to avoid calling unresponsive nodes.\n\n### Public API endpoints\n\n```\nGET  /api/v1/get/:id\nGET  /api/v1/meta/:id\nPOST /api/v1/put\nPOST /api/v1/delete/:id\nGET  /api/v1/id\nGET  /api/v1/version\nGET  /api/v1/ping\nGET  /api/v1/stats\n```\n\n### How to upload files\n\n1. **Generate a new ID.** All files are associated with IDs of [ULID](https://github.com/oklog/ulid) format, so you must generate your own or just ask any node for new ID.\n```bash\n$ curl localhost:10999/api/v1/id\n\n01BRNMMS1DK3CBD4ZZM2TQ8C5B\n```\n\n2. **Choose consistency level** Three levels are available:\n```go\n    // ConsistencyLocal flags file for local persistence only, implying\n    // that the file body will be stored on a single node. Default.\n    ConsistencyLocal ConsistencyLevel = 0\n    // ConsistencyS3 flags file for local+S3 persistence, implying that the file\n    // body will be stored on a single node and Amazon S3.\n    ConsistencyS3 ConsistencyLevel = 1\n    // ConsistencyFull flags file to be replicated across all existing nodes in cluster and S3.\n    ConsistencyFull ConsistencyLevel = 2\n```\n\n3. **Specify headers** The following headers are available:\n    * `X-Meta-ID` is a previously generated or retrieved [ULID](https://github.com/oklog/ulid);\n    * `X-Meta-Name` is the file name, used with extension to serve the content with proper type;\n    * `X-Meta-ConsistencyLevel` specifies the consistency level for the file, it may be upgraded later;\n    * `X-Meta-UserMeta` specifies any meta data for the file as JSON map, stored in S3 tags.\n\n4. **POST** Example, let's upload `test.txt` with replication across cluster and S3.\n\n```\n$ curl -d @test.txt -H \"X-Meta-ConsistencyLevel: 2\" -H \"X-Meta-Name: test.txt\" \\\n    -H \"X-Meta-ID: 01BRNMMS1DK3CBD4ZZM2TQ8C5B\" localhost:10999/api/v1/put\n```\n\n### How to read files\n\nAccessing a file is straightforward:\n\n```\n$ curl localhost:10999/api/v1/get/01BRNMMS1DK3CBD4ZZM2TQ8C5B\n\nIt works!\n```\n\nMore is going on under the covers. Apparently the file exists on both nodes and S3, but in case when file is stored only on a single node (with level=1), then the node would fetch this file from another one and serve, this fact can be checked by inspecting the headers:\n\n```\n$ curl -v localhost:10999/api/v1/get/01BRNMMS1DK3CBD4ZZM2TQ8C5B\n\n\u003c HTTP/1.1 200 OK\n\u003c Accept-Ranges: bytes\n\u003c Content-Length: 9\n\u003c Content-Type: text/plain; charset=utf-8\n\u003c Last-Modified: Tue, 29 Aug 2017 00:11:35 GMT\n\u003c X-Meta-Consistencylevel: 2\n\u003c X-Meta-Id: 01BRNMMS1DK3CBD4ZZM2TQ8C5B\n\u003c X-Meta-Name: test.txt\n\u003c Date: Tue, 29 Aug 2017 00:19:45 GMT\n```\n\nIn case when file is fetched from another node, `X-Meta-Symlink: true` will appear. If file is known but has been deleted, a 404 with `X-Meta-Deleted: true` will be served. And if file has been missing locally and fetched from S3, a header flag `X-Meta-Fetched: true` will appear.\n\nBy default, fetching from S3 in case of cache misses is disabled, to get a file with possibility of fetching it from the cloud storage backend, use `X-Meta-Fetch`:\n\n```bash\n# to check we delete local state and restarted nodes (!)\n\n$ curl -v localhost:10999/api/v1/get/01BRNMMS1DK3CBD4ZZM2TQ8C5B\n\n\u003c HTTP/1.1 404 Not Found\n\n$ curl -v -H \"X-Meta-Fetch: 1\" localhost:10999/api/v1/get/01BRNMMS1DK3CBD4ZZM2TQ8C5B\n\n\u003c HTTP/1.1 200 OK\n\u003c Accept-Ranges: bytes\n\u003c Content-Length: 9\n\u003c Content-Type: text/plain; charset=utf-8\n\u003c Last-Modified: Tue, 29 Aug 2017 00:49:17 GMT\n\u003c X-Meta-Consistencylevel: 2\n\u003c X-Meta-Fetched: true\n\u003c X-Meta-Id: 01BRNMMS1DK3CBD4ZZM2TQ8C5B\n\u003c X-Meta-Name: test.txt\n\u003c Date: Tue, 29 Aug 2017 00:49:17 GMT\n\nIt works!\n```\n\nNotice that file has been fetched with `X-Meta-Fetched: true`, it also has all properties saved such as name, content type and the consistency level. The latter means it was also replicated again across the nodes.\n\n## Acknowledgements\n\nThe project is in Open Beta stage, please test it before using in something serious.\n\n## TODO / Roadmap\n\n* Implement eviction policy (LRU / ARC / ...)\n* Document the internal design\n* Improve deployment scripts\n* Test coverage\n* Implement a sample client\n* Benchmarks\n\n## License\n\nMIT\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspheresoftware%2Fobjstore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspheresoftware%2Fobjstore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspheresoftware%2Fobjstore/lists"}