{"id":18300505,"url":"https://github.com/eclipse-zenoh/zenoh-backend-s3","last_synced_at":"2025-04-07T13:06:12.114Z","repository":{"id":65471372,"uuid":"550122393","full_name":"eclipse-zenoh/zenoh-backend-s3","owner":"eclipse-zenoh","description":"Backend and Storages for Zenoh using AWS S3 API","archived":false,"fork":false,"pushed_at":"2025-04-03T08:41:27.000Z","size":1195,"stargazers_count":19,"open_issues_count":1,"forks_count":10,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-04-03T09:32:23.890Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eclipse-zenoh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-12T08:36:44.000Z","updated_at":"2025-04-03T08:41:32.000Z","dependencies_parsed_at":"2023-11-28T10:27:30.509Z","dependency_job_id":"8fcb9345-53b3-4260-b820-f66e039b6291","html_url":"https://github.com/eclipse-zenoh/zenoh-backend-s3","commit_stats":{"total_commits":86,"total_committers":4,"mean_commits":21.5,"dds":0.2325581395348837,"last_synced_commit":"0e36d31edf411999cb19f146cde634d66e36d1ac"},"previous_names":[],"tags_count":91,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eclipse-zenoh%2Fzenoh-backend-s3","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eclipse-zenoh%2Fzenoh-backend-s3/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eclipse-zenoh%2Fzenoh-backend-s3/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eclipse-zenoh%2Fzenoh-backend-s3/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eclipse-zenoh","download_url":"https://codeload.github.com/eclipse-zenoh/zenoh-backend-s3/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247657276,"owners_count":20974344,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-05T15:12:41.184Z","updated_at":"2025-04-07T13:06:12.092Z","avatar_url":"https://github.com/eclipse-zenoh.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"https://raw.githubusercontent.com/eclipse-zenoh/zenoh/main/zenoh-dragon.png\" height=\"150\"\u003e\n\n[![Discussion](https://img.shields.io/badge/discussion-on%20github-blue)](https://github.com/eclipse-zenoh/roadmap/discussions)\n[![Discord](https://img.shields.io/badge/chat-on%20discord-blue)](https://discord.gg/vSDSpqnbkm)\n[![License](https://img.shields.io/badge/License-EPL%202.0-blue)](https://choosealicense.com/licenses/epl-2.0/)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n\n# Eclipse Zenoh\n\nThe Eclipse Zenoh: Zero Overhead Pub/sub, Store/Query and Compute.\n\nZenoh (pronounce _/zeno/_) unifies data in motion, data at rest and computations. It carefully blends traditional pub/sub with geo-distributed storages, queries and computations, while retaining a level of time and space efficiency that is well beyond any of the mainstream stacks.\n\nCheck the website [zenoh.io](http://zenoh.io) and the [roadmap](https://github.com/eclipse-zenoh/roadmap) for more detailed information.\n\n# S3 backend\n\nIn zenoh a backend is a storage technology (such as DBMS, time-series database, file system...) alowing to store the\nkeys/values publications made via zenoh and return them on queries.\nSee the [zenoh documentation](https://zenoh.io/docs/manual/plugin-storage-manager/#backends-and-volumes) for more details.\n\nThis backend relies on [Amazon S3](https://aws.amazon.com/s3/?nc1=h_ls) to implement the storages. It is also compatible to work with [MinIO](https://min.io/) object storage.\n\nIts library name (without OS specific prefix and extension) that zenoh will rely on to find it and load it is **`libzenoh_backend_s3`**.\n\n:point_right: **Install latest release:** see [below](#how-to-install-it)\n\n:point_right: **Build \"main\" branch:** see [below](#how-to-build-it)\n\n---\n\n## **Examples of usage**\n\nPrerequisites:\n\n- You have a zenoh router (`zenohd`) installed, and the `libzenoh_backend_s3` library file is available in `~/.zenoh/lib`. Alternatively we can set a symlink to the library, for instance by running:\n\n  ```bash\n  ln -s ~/zenoh-backend-s3/target/release/libzenoh_backend_s3.dylib  ~/.zenoh/lib/libzenoh_backend_s3.dylib\n  ```\n\n- You have an S3 instance running, this could be an AmazonS3 instance or a MinIO instance.\n\nYou can setup storages either at zenoh router startup via a configuration file, either at runtime via the zenoh admin space, using for instance the REST API (see [https://zenoh.io/docs/manual/plugin-storage-manager/](https://zenoh.io/docs/manual/plugin-storage-manager/)).\n\n### Setting up a MinIO instance\n\nIn order to run the examples of usage from the following section, it is convenient to launch a MinIO instance. To launch MinIO on a Docker container you first, install MinIO with\n\n```bassh\ndocker pull minio/minio\n```\n\nAnd then you can use the following command to launch the instance:\n\n```bash\ndocker run -p 9000:9000 -p 9090:9090  --user $(id -u):$(id -g)  --name minio -e 'MINIO_ROOT_USER=AKIAIOSFODNN7EXAMPLE' -e 'MINIO_ROOT_PASSWORD=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY'  -v ${HOME}/minio/data:/data   quay.io/minio/minio server data --console-address ':9090'\n```\n\nIf successful, then the console can be accessed on [http://localhost:9090](http://localhost:9090).\n\n### **Setup via a JSON5 configuration file**\n\n- Create a `zenoh.json5` configuration file containing:\n\n  ```json5\n  {\n    plugins: {\n      // Configuration of \"storage_manager\" plugin:\n      storage_manager: {\n        volumes: {\n          s3: {\n            // AWS region to which connect (see https://docs.aws.amazon.com/general/latest/gr/s3.html).\n            // This field is mandatory if you are going to communicate with an AWS S3 server and\n            // optional in case you are working with a MinIO S3 server.\n            region: \"eu-west-1\",\n\n            // Endpoint where the S3 server is located.\n            // This parameter allows you to specify a custom endpoint when working with a MinIO S3\n            // server.\n            // This field is mandatory if you are working with a MinIO server and optional in case\n            // you are working with an AWS S3 server as long as you specified the region, in which\n            // case the endpoint will be resolved automatically.\n            url: \"https://s3.eu-west-1.amazonaws.com\",\n\n            // Optional TLS specific parameters to enable HTTPS with MinIO. Configuration shared by\n            // all the associated storages.\n            // tls: {\n            //  private: {\n            //    // Certificate authority to authenticate the server.\n            //    root_ca_certificate_file: \"/home/user/certificates/minio/ca.pem\",\n            //\n            //    // Alternatively you can inline your certificate encoded with base 64:\n            //    root_ca_certificate_base64: \"\u003cYOUR_CERTIFICATE_ENCODED_WITH_BASE64\u003e\"\n            //  }\n            //},\n          },\n        },\n        storages: {\n          // Configuration of a \"demo\" storage using the S3 volume. Each storage is associated to a\n          // single S3 bucket.\n          s3_storage: {\n            // The key expression this storage will subscribes to\n            key_expr: \"s3/example/*\",\n\n            // this prefix will be stripped from the received key when converting to database key.\n            // i.e.: \"demo/example/a/b\" will be stored as \"a/b\"\n            strip_prefix: \"s3/example\",\n\n            volume: {\n              // Id of the volume this storage is associated to\n              id: \"s3\",\n\n              // Bucket to which this storage is associated to\n              bucket: \"zenoh-bucket\",\n\n              // The storage attempts to create the bucket, but if the bucket already exists and is\n              // owned by you, then with 'reuse_bucket' you can associate that preexisting bucket to\n              // the storage, otherwise it will fail.\n              reuse_bucket: true,\n\n              // If the storage is read only, it will only handle GET requests\n              read_only: false,\n\n              // strategy on storage closure, either `destroy_bucket` or `do_nothing`\n              on_closure: \"destroy_bucket\",\n\n              private: {\n                // Credentials for interacting with the S3 bucket\n                access_key: \"\u003cYOUR ACCESS KEY\u003e\",\n                secret_key: \"\u003cYOUR SECRET KEY\u003e\",\n              },\n            },\n          },\n        },\n      },\n      // Optionally, add the REST plugin\n      rest: { http_port: 8000 },\n    },\n  }\n  ```\n\n- Run the zenoh router with:\n\n  ```bash\n  zenohd -c zenoh.json5\n  ```\n\n#### Volume configuration when working with AWS S3 storage\n\nWhen working with the AWS S3 storage, the region must be specified following the region names indicated in the [Amazon Simple Storage Service endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/s3.html) documentation. The url of the endpoint is not required as the internal endpoint resolver will automatically\nfind which one is the endpoint associated to the region specified.\n\nAll the storages associated to the volume will use the same region.\n\nThe volumes section on the config file will look like:\n\n```json\nstorage_manager {\n  volumes: {\n    s3: {\n        // AWS region to which connect\n        region: \"eu-west-1\",\n    }\n  },\n  ...\n}\n```\n\n#### Volume configuration when working with MinIO\n\nInversely, when working with a MinIO S3 storage, then we need to specify the endpoint of the storage rather than the region, which will be ignored by the MinIO server. We can save ourselves to specify the region in that case.\n\nThe volumes section on the config file will look like:\n\n```json\nstorage_manager {\n  volumes: {\n    s3: {\n        url: \"http://localhost:9000\",\n    }\n  },\n  ...\n}\n```\n\n### **Setup at runtime via `curl` commands on the admin space**\n\n- Run the zenoh router:\n\n  ```bash\n  cargo run --bin=zenohd\n  ```\n\n- Add the \"s3\" backend (the \"zenoh_backend_s3\" library will be loaded):\n\n  ```bash\n  curl -X PUT -H 'content-type:application/json' -d '{url: \"http://localhost:9000\", private: {access_key: \"AKIAIOSFODNN7EXAMPLE\", secret_key: \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\"}}' http://localhost:8000/@/router/local/config/plugins/storage_manager/volumes/s3\n  ```\n\n- Add the \"s3_storage\" storage using the \"s3\" backend:\n\n  ```bash\n  curl -X PUT -H 'content-type:application/json' -d '{key_expr:\"s3/example/*\", strip_prefix:\"s3/example\", volume: {id: \"s3\", bucket: \"zenoh-bucket\", create_bucket: true, region: \"eu-west-3\", on_closure: \"do_nothing\", private: {access_key: \"AKIAIOSFODNN7EXAMPLE\", secret_key: \"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\"}}}' http://localhost:8000/@/router/local/config/plugins/storage_manager/storages/s3_storage\n  ```\n\n### **Tests using the REST API**\n\nUsing `curl` to publish and query keys/values, you can:\n\n```bash\n# Put values that will be stored in the S3 storage\ncurl -X PUT -H 'content-type:application/json' -d '{\"example_key\": \"example_value\"}' http://0.0.0.0:8000/s3/example/test\n\n# To get the stored object\ncurl -X GET -H {} -d '{}' http://0.0.0.0:8000/s3/example/test\n\n# To delete the previous object\ncurl -X DELETE -H {} -d '{}' http://0.0.0.0:8000/s3/example/test\n\n# To delete the whole storage and the bucket if configured (note in order for this test to work, you need to setup adminspace read/write permissions)\ncurl -X DELETE 'http://0.0.0.0:8000/@/router/local/config/plugins/storage_manager/storages/s3_storage'\n\n# To delete the whole volume (note in order for this test to work, you need to setup adminspace read/write permissions)\ncurl -X DELETE 'http://0.0.0.0:8000/@/router/local/config/plugins/storage_manager/volumes/s3'\n```\n\n## **Enabling TLS on MinIO**\n\nIn order to establish secure communication through HTTPS we need to provide a certificate of the certificate authority that validates the server credentials.\n\nTLS certificates can be generated as explained in the [zenoh documentation using Minica](https://zenoh.io/docs/manual/tls/). When running\n\n```bash\nminica --domains localhost\n```\n\na private key, a public certificate and a certificate authority certificate is generated:\n\n```raw\n└── certificates\n    ├── localhost\n    │   ├── cert.pem\n    │   └── key.pem\n    ├── minica-key.pem\n    └── minica.pem\n```\n\nOn the config file, we need to specify the `root_ca_certificate_file` as this will allow the s3 plugin to validate the MinIO server keys.\nExample:\n\n```json\ntls: {\n  private: {\n    root_ca_certificate_file: \"/home/user/certificates/minio/minica.pem\",\n  },\n},\n```\n\nHere, the `root_ca_certificate_file` corresponds to the generated _minica.pem_ file.\nYou can also embed directly the root_ca_certificate by inlining it under the filed `root_ca_certificate_base64`, encoded with base64.\n\nThe _cert.pem_ and _key.pem_ files correspond to the public certificate and private key respectively. We need to rename them as _public.crt_ and _private.key_ respectively and store them under the MinIO configuration directory (as specified in the [MinIO documentation](https://min.io/docs/minio/linux/operations/network-encryption.html#enabling-tls)). In case you are using running a docker container as previously shown, then we will need to mount the folder containing the certificates as a volume; supposing we stored our certificates under `${HOME}/minio/certs`, we need to start our container as follows:\n\n```bash\ndocker run -p 9000:9000 -p 9090:9090  --user $(id -u):$(id -g)  --name minio -e 'MINIO_ROOT_USER=AKIAIOSFODNN7EXAMPLE' -e 'MINIO_ROOT_PASSWORD=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY' -v ${HOME}/minio/data:/data -v ${HOME}/minio/certs:/certs quay.io/minio/minio server data --certs-dir certs --console-address ':9090'\n```\n\nFinally the volume configuration should then look like:\n\n```json\nstorage_manager: {\n  volumes: {\n    s3: {\n        // Endpoint where the S3 server is located\n        url: \"https://localhost:9000\",\n\n        // Configure TLS specific parameters\n        tls: {\n          private: {\n            root_ca_certificate_file: \"/home/user/certificates/minio_certs/minica.pem\",\n          },\n        },\n    },\n  },\n```\n\n_Note: do not forget to modify the endpoint protocol, for instance from `http://localhost:9090` to `https://localhost:9090`_\n\n---\n\n## How to install it\n\nTo install the latest release of this backend library, you can do as follows:\n\n### Manual installation (all platforms)\n\nAll release packages can be downloaded from:\n\n- [https://download.eclipse.org/zenoh/zenoh-backend-s3/latest/](https://download.eclipse.org/zenoh/zenoh-backend-s3/latest/)\n\nEach subdirectory has the name of the Rust target. See the platforms each target corresponds to on [https://doc.rust-lang.org/stable/rustc/platform-support.html](https://doc.rust-lang.org/stable/rustc/platform-support.html)\n\nChoose your platform and download the `.zip` file.  \nUnzip it in the same directory than `zenohd` or to any directory where it can find the backend library (e.g. /usr/lib or ~/.zenoh/lib)\n\n### Linux Debian\n\nAdd Eclipse Zenoh private repository to the sources list, and install the `zenoh-backend-s3` package:\n\n```bash\necho \"deb [trusted=yes] https://download.eclipse.org/zenoh/debian-repo/ /\" | sudo tee -a /etc/apt/sources.list \u003e /dev/null\nsudo apt update\nsudo apt install zenoh-backend-s3\n```\n\n---\n\n## How to build it\n\n\u003e :warning: **WARNING** :warning: : Zenoh and its ecosystem are under active development. When you build from git, make sure you also build from git any other Zenoh repository you plan to use (e.g. binding, plugin, backend, etc.). It may happen that some changes in git are not compatible with the most recent packaged Zenoh release (e.g. deb, docker, pip). We put particular effort in mantaining compatibility between the various git repositories in the Zenoh project.\n\nAt first, install [Cargo and Rust](https://doc.rust-lang.org/cargo/getting-started/installation.html). If you already have the Rust toolchain installed, make sure it is up-to-date with:\n\n```bash\nrustup update\n```\n\n\u003e :warning: **WARNING** :warning: : As Rust doesn't have a stable ABI, the backend library should be\n\u003e built with the exact same Rust version than `zenohd`. Otherwise, incompatibilities in memory mapping\n\u003e of shared types between `zenohd` and the library can lead to a `\"SIGSEV\"` crash.\n\nTo know the Rust version you're `zenohd` has been built with, use the `--version` option.  \nExample:\n\n```bash\n$ zenohd --version\nzenohd v0.7.0-rc-365-geca888b4-modified built with rustc 1.69.0 (84c898d65 2023-04-16)\nThe zenoh router v0.7.0-rc-365-geca888b4-modified built with rustc 1.69.0 (84c898d65 2023-04-16)\n```\n\nHere, `zenohd` has been built with the rustc version `1.69.0`.\nInstall and use this toolchain with the following command:\n\n```bash\nrustup default 1.69.0\n```\n\nAnd then build the backend with:\n\n```bash\ncargo build --release --all-targets\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feclipse-zenoh%2Fzenoh-backend-s3","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feclipse-zenoh%2Fzenoh-backend-s3","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feclipse-zenoh%2Fzenoh-backend-s3/lists"}