{"id":13671718,"url":"https://github.com/s3d-rs/s3d","last_synced_at":"2025-04-27T18:31:37.270Z","repository":{"id":40366025,"uuid":"411819282","full_name":"s3d-rs/s3d","owner":"s3d-rs","description":"s3d is an S3 daemon for the Edge written in Rust","archived":true,"fork":false,"pushed_at":"2022-05-12T16:04:51.000Z","size":1085,"stargazers_count":42,"open_issues_count":4,"forks_count":12,"subscribers_count":12,"default_branch":"main","last_synced_at":"2024-11-11T09:44:03.169Z","etag":null,"topics":["rust","s3","s3d"],"latest_commit_sha":null,"homepage":"https://s3d.rs","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/s3d-rs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-09-29T20:24:01.000Z","updated_at":"2023-12-28T20:13:04.000Z","dependencies_parsed_at":"2022-08-09T18:23:24.016Z","dependency_job_id":null,"html_url":"https://github.com/s3d-rs/s3d","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s3d-rs%2Fs3d","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s3d-rs%2Fs3d/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s3d-rs%2Fs3d/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s3d-rs%2Fs3d/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/s3d-rs","download_url":"https://codeload.github.com/s3d-rs/s3d/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251187253,"owners_count":21549610,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["rust","s3","s3d"],"created_at":"2024-08-02T09:01:17.061Z","updated_at":"2025-04-27T18:31:36.037Z","avatar_url":"https://github.com/s3d-rs.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"\u003cdiv id=\"top\"\u003e\u003c/div\u003e\n\u003cdiv align=\"center\"\u003e\n  \u003c!-- GItHUB --\u003e\n  \u003ca alt=\"github\" href=\"https://github.com/s3d-rs/s3d\"\u003e\n    \u003cimg src=\"https://img.shields.io/github/languages/code-size/s3d-rs/s3d?label=github\u0026logo=github\u0026color=lightblue\" /\u003e\n  \u003c/a\u003e\n  \u003ca alt=\"build\" href=\"https://github.com/s3d-rs/s3d/actions\"\u003e\n    \u003c!-- \u003cimg src=\"https://github.com/s3d-rs/s3d/workflows/build/badge.svg\" /\u003e --\u003e\n    \u003cimg src=\"https://img.shields.io/github/workflow/status/s3d-rs/s3d/build/main?color=lightgreen\" /\u003e\n  \u003c/a\u003e\n  \u003c!--\n  \u003ca alt=\"releases\" href=\"https://github.com/s3d-rs/s3d/releases/latest\"\u003e\n    \u003cimg src=\"https://img.shields.io/github/v/release/s3d-rs/s3d\" /\u003e\n  \u003c/a\u003e\n  --\u003e\n  \u003c!-- RUST --\u003e\n  \u003ca alt=\"crate\" href=\"https://crates.io/crates/s3d\"\u003e\n    \u003cimg src=\"https://img.shields.io/crates/v/s3d.svg?logo=rust\u0026logoColor=white\u0026color=lightgrey\" /\u003e\n  \u003c/a\u003e\n  \u003c!--\n  \u003ca alt=\"s3d at docs.rs\" href=\"http://docs.rs/s3d\"\u003e\n    \u003cimg src=\"https://docs.rs/s3d/badge.svg\" /\u003e\n  \u003c/a\u003e\n  --\u003e\n  \u003c!-- OTHER --\u003e\n  \u003ca alt=\"discord\" href=\"https://discord.com/channels/897764851580035072\"\u003e\n    \u003cimg src=\"https://img.shields.io/discord/897764851580035072?logo=discord\u0026logoColor=white\u0026label=discord\u0026color=lightgreen\" /\u003e\n  \u003c/a\u003e\n  \u003ca alt=\"license\" href=\"LICENSE\"\u003e\n    \u003cimg src=\"https://img.shields.io/github/license/s3d-rs/s3d?color=lightgreen\" /\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\u003cbr /\u003e\n\u003cdiv align=\"center\" style=\"background-color: hsla(0,0,0%,0.1); text-align: center\"\u003e\n  \u003ca alt=\"s3d logo\" href=\"https://s3d.rs\" style=\"background-color: hsla(0,0,0%,0.1); text-align: center\"\u003e\n    \u003cimg alt=\"s3d\" src=\"s3d.png\" width=\"250\" /\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\n# The S3 Daemon\n\n`s3d` is a daemon for data access using S3 API. A modern cousin of `nfsd`, `ftpd`, `httpd`, etc. It is designed to be simple, tiny, blazing fast, and portable in order to fit in a variety of environments from developer machines, containers, kubernetes, edge devices, etc.\n\nBy default, `s3d` serves the S3 API as a gateway to a main remote S3 storage (AWS/compatible), with ultimate protocol compatiblity (based on the AWS SDK and Smithy API), it adds critical features needed by remote clients such as queueing, caching, and synching, in order to optimize performance, increase data availability, and service continuity for its clients.\n\nThe need for a daemon running alongside applications emerges in Edge computing use cases, where data is stored and processed locally at the edge as it gets collected, while some of the data gets synced to and from a main data storage. Refer to [Wikipedia - Edge computing](https://en.wikipedia.org/wiki/Edge_computing): _\"Edge computing is a distributed computing paradigm that brings computation and data storage closer to the sources of data. This is expected to improve response times and save bandwidth. ...\"_.\n\n# Features\n\nMaturity legend: 🥉 - design phase, 🥈 - development phase, 🥇 - released.\n\n- 🥈 \u0026nbsp; **S3 API**\n  - Generated S3 protocol code with [awslabs/smithy-rs](https://github.com/awslabs/smithy-rs) which builds the AWS SDK for Rust.\n  - Provides compatible parsing of API operations, inputs, outputs, errors, and the server skeleton.\n- 🥈 \u0026nbsp; **Write Queue**\n  - Writing new objects to local filesystem first to tolerate connection issues.\n  - Pushing in the background to remote storage.\n- 🥉 \u0026nbsp; **Read Cache**\n  - Store cached and prefetched objects in local filesystem.\n  - Reduce egress costs and latency of reads from remote storage.\n- 🥉 \u0026nbsp; **Filters**\n  - Fine control over which objects to include/exclude for upload/cache/sync.\n  - Filter by bucket name, bucket tags, object keys (or prefixes), object tags, and object meta-data.\n  - Optional integration with [OpenPolicyAgent (OPA)](https://www.openpolicyagent.org) for advanced filtering.\n- 🥉 \u0026nbsp; **Sync Folder**\n  - Continuous, bidirectional and background sync of local dirs to remote buckets.\n  - This is a simple way to get data that begins as local files to the remote S3 storage.\n- 🥉 \u0026nbsp; **Fuse Mount**\n  - Virtual filesystem mountpoint provided by the daemon (see [kernel fuse docs](https://www.kernel.org/doc/html/latest/filesystems/fuse.html)).\n  - The filesystem can be accessed normally for applications that do not use the S3 API.\n- 🥉 \u0026nbsp; **Metrics**\n  - Optional integration with [OpenTelemetry](https://opentelemetry.io).\n  - Expose logging, metrics, and tracing of S3 operations and `s3d` features.\n\n# Docs\n\n- 🧑‍🚀 \u0026nbsp; **[User guide](docs/user-guide.md)** - how to use features and configurations.\n- 🥷 \u0026nbsp; **[Developer guide](docs/developer-guide.md)** - how to build and test.\n- 🧝 \u0026nbsp; **[Architecture page](docs/architecture.md)** - designs of components, software and roadmap ideas.\n\n# Status\n\n🧪🧪❕ \u0026nbsp; **Experimental** \\\n🧪🧪❕ \u0026nbsp; \\\n🧪🧪❕ \u0026nbsp; This project is still in it's early days, which means it's a great time \\\n🧪🧪❕ \u0026nbsp; to affect its direction, and it welcomes contributions and open discussions. \\\n🧪🧪❕ \u0026nbsp; \\\n🧪🧪❕ \u0026nbsp; Keep in mind that all internal and external interfaces are considered unstable \\\n🧪🧪❕ \u0026nbsp; and might change without notice.\n\n# Getting Started\n\nUntil the first releases are available, please refer to the [Developer guide](docs/developer-guide.md) for building `s3d` from source.\n\nHere are some commands to explore:\n\n```bash\nmake                        # build from source (see dev guide for prerequisites)\neval $(make env)            # defines aliases such as s3d -\u003e build output binary\naws configure list          # make sure remote S3 credentials are configured\ns3d run                     # runs daemon (foreground)\ns3d status                  # check daemon status\ns3d status bucket/key       # check bucket or object status\ns3d ls bucket/prefix        # list buckets or objects\ns3d get bucket/key \u003e file   # get object data to stdout (meta-data to stderr)\ns3d put bucket/key \u003c file   # put object data from stdin\ns3d set bucket/key          \\\n  --tag s3d.upload=true     \\\n  --tag s3d.cache=pin       # set tags for bucket or object to be used in filters\ns3d \u003ccommand\u003e --help        # show usage for command\n```\n\n# Project\n\n- [Github repo](https://github.com/s3d-rs/s3d) - The project lives here, set your \u003cbutton\u003e👁️  Watch\u003c/button\u003e options to get notified on releases, issues, etc.\n- [Discord chat](https://discord.com/channels/897764851580035072) - use this [invite link](https://discord.gg/kPWHDuCdhh) to join.\n- [Redhat-et](https://github.com/redhat-et) - this project was initiated at Red Hat Emerging Technologies.\n- [License](LICENSE) - Apache 2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fs3d-rs%2Fs3d","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fs3d-rs%2Fs3d","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fs3d-rs%2Fs3d/lists"}