{"id":18497449,"url":"https://github.com/elastio/ssstar","last_synced_at":"2025-04-06T14:12:58.805Z","repository":{"id":58301041,"uuid":"523466576","full_name":"elastio/ssstar","owner":"elastio","description":"tar-like utility for constructing archives of objects stored on S3 or S3-compatible services","archived":false,"fork":false,"pushed_at":"2025-03-27T12:56:21.000Z","size":721,"stargazers_count":22,"open_issues_count":21,"forks_count":4,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-30T13:09:10.767Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/elastio.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-10T19:08:28.000Z","updated_at":"2025-03-02T22:12:31.000Z","dependencies_parsed_at":"2023-12-31T22:48:18.023Z","dependency_job_id":"d0aa0cba-3084-4009-b60d-51cfd9536c29","html_url":"https://github.com/elastio/ssstar","commit_stats":{"total_commits":154,"total_committers":4,"mean_commits":38.5,"dds":"0.12337662337662336","last_synced_commit":"78191c164ac1c4d317d7fd84fd0d9b11943cadcc"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastio%2Fssstar","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastio%2Fssstar/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastio%2Fssstar/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastio%2Fssstar/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/elastio","download_url":"https://codeload.github.com/elastio/ssstar/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247492565,"owners_count":20947545,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T13:34:35.377Z","updated_at":"2025-04-06T14:12:58.779Z","avatar_url":"https://github.com/elastio.png","language":"Rust","readme":"## ssstar\n\nssstar is a Rust library crate as well as a command-line tool to create and extract `tar`-compatible archives containing\nobjects stored in S3 or S3-compatible storage. It works similarly to GNU `tar`, and produces archives that are 100%\ncompatible with `tar`, though it uses different command line arguments.\n\n[![Crates.io](https://img.shields.io/crates/v/ssstar.svg)](https://crates.io/crates/ssstar)\n[![Docs.rs](https://docs.rs/ssstar/badge.svg)](https://docs.rs/ssstar)\n[![CI](https://github.com/elastio/ssstar/workflows/CI/badge.svg)](https://github.com/elastio/ssstar/actions)\n[![Coverage Status](https://coveralls.io/repos/github/elastio/ssstar/badge.svg?branch=master)](https://coveralls.io/github/elastio/ssstar?branch=master)\n![Crates.io](https://img.shields.io/crates/l/ssstar)\n\n`ssstar` provides a cross-platform Rust-powered CLI as well as a Rust library crate that lets you create tar archives\ncontaining objects from S3 and S3-compatible object storage, regardless of size. `ssstar` applies concurrency\naggressively, and uses a streaming design which means even multi-TB objects can be processed with minimal memory\nutilization. The resulting tar archive can itself be uploaded to object storage, written to a local file, or written\nto `stdout` and piped to another command line tool.\n\nWe built `ssstar` so our customers using the `elastio` cloud native backup and recovery CLI could backup and restore S3\nbuckets directly into and from [Elastio](https://elastio.com/) vaults, however we made the tool generic enough that it can be used by itself\nwhenever you need to package one or more S3 objects into a tarball.\n\n## Installation\n\n### Cargo\n\nOn any supported platform (meaning Windows, macOS (both Intel and Apple Silicon) and Linux), if you have a recent Rust\ncompiler installed you can use `cargo install` to get the ssstar CLI:\n\n- Ensure you have at least Rust 1.63.0 installed by following [this](https://www.rust-lang.org/tools/install) guide.\n- Run `cargo install ssstar-cli --locked` to compile `ssstar` from source and install locally.\n\n### Precompiled binaries\n\nSee the [GitHub Releases](https://github.com/elastio/ssstar/releases/latest) for pre-compiled binaries for Windows, mac, and Linux.\n\n## Usage (without Elastio)\n\nTo create a tar archive, you specify S3 buckets, objects, entire prefixes, or globs, as well as where you want the tar\narchive to be written:\n\n```shell\n\n# Archive an entire bucket and write the tar archive to another bucket\nssstar create \\\n  s3://my-source-bucket \\\n  --s3 s3://my-destination-bucket/backup.tar\n\n# Archive all objects in the `foo/` prefix (non-recursive) and write the tar archive to a local file\nssstar create \\\n  s3://my-source-bucket/foo/ \\\n  --file ./backup.tar\n\n# Archive some specific objects identified by name, and write the tar archive to stdout and pipe that to\n# gzip\nssstar create \\\n  s3://my-source-bucket/object1 s3://my-source-bucket/object2 \\\n  --stdout | gzip \u003e backup.tar.gz\n\n# Archive all objects matching a glob, and write the tar archive to another bucket\nssstar create \\\n  \"s3://my-source-bucket/foo/**\" \\\n  --s3 s3://my-destination-bucket/backup.tar\n```\n\nYou can pass multiple inputs to `ssstar create`, using a mix of entire buckets, prefixes, specific objects, and globs.\nJust make sure that when you use globs you wrap them in quotes, otherwise your shell may try to evaluate them. For\nexample:\n\n```shell\n# Archive a bunch of different inputs, writing the result to a file\nssstar create \\\n  s3://my-source-bucket/                  \\ # \u003c-- include all objects in `my-source-bucket`\n  s3://my-other-bucket/foo/               \\ # \u003c-- include all objects in `foo/` (non-recursive)\n  s3://my-other-bucket/bar/boo            \\ # \u003c-- include the object with key `bar/boo`\n  \"s3://yet-another-bucket/logs/2022*/**\" \\ # \u003c-- recursively include all objects in any prefix `logs/2022*`\n  --file ./backup.tar                     # \u003c-- this is the path where the tar archive will be written\n```\n\nTo extract a tar archive and write the contents directly to S3 objects, you specify where to find the tar archive,\noptional filters to filter what is extracted, and the S3 bucket and prefix to which to extract the contents.\n\nA simple example:\n\n```shell\n# Extract a local tar archive to the root of an S3 bucket `my-bucket`\nssstar extract --file ./backup.tar s3://my-bucket\n```\n\nEach file in the tar archive will be written to the bucket `my-bucket`, with the object key equal to the file path\nwithin the archive. For example if the archive contains a file `foo/bar/baz.txt`, that file will be written to\n`s3://my-bucket/foo/bar/baz.txt`.\n\nYou can provide not just a target bucket but also a prefix as well, e.g.:\n\n```shell\n# Extract a local tar archive to the prefix `restored/` of an S3 bucket `my-bucket`\nssstar extract --file ./backup.tar s3://my-bucket/restored/\n```\n\nIn that case, if the tar archive contains a file `foo/bar/baz.txt`, it will be written to\n`s3://my-bucket/restored/foo/bar/baz.txt`. _NOTE_: In S3, prefixes don't necessarily end in `/`; if you don't provide\nthe trailing `/` character to the S3 URL passed to `ssstar extract`, it will not be added for you! Instead you'll get\nsomething like `s3://my-bucket/restoredfoo/bar/baz`, which may or may not be what you actually want!\n\nIf you don't want to extract the full contents of the archive, you can specify one or more filters. These can be exact\nfile paths, directory paths ending in `/`, or globs. For example:\n\n```shell\nssstar extract --file ./backup.tar \\\n  foo/bar/baz.txt          \\ # \u003c-- extract the file `foo/bar/baz.txt` if it's present in the archive\n  boo/                     \\ # \u003c-- extract all files in the `boo` directory (recursive)\n  \"baz/**/*.txt\"           \\ # \u003c-- extract any `.txt` file anywhere in `baz/`, recursively\n  s3://my-bucket/restored/   # \u003c-- write all matching files to the `restored/` prefix in `my-bucket`\n```\n\n## Usage (with Elastio)\n\nTo use with Elastio, create archives with the `--stdout` option and pipe to `elastio stream backup`, and restore them by piping\n`elastio stream restore` to `ssstar extract` with the `--stdin` option. For example:\n\n```shell\n# Backup an entire S3 bucket `my-source-bucket` to the default Elastio vault:\nssstar create s3://my-source-bucket/ --stdout \\\n  | elastio stream backup --hostname-override my-source-bucket --stream-name my-backup\n```\n\n```shell\n# Restore a recovery point with ID `$RP_ID` from Elastio to the `my-destination-bucket` bucket:\nelastio stream restore --rp $RP_ID \\\n  | ssstar extract --stdin s3://my-destination-bucket\n```\n\nFor more about using the Elastio CLI, see the [Elastio CLI docs](https://docs.elastio.com/src/elastio-cli/elastio-cli-overview.html)\n\n## Advanced CLI Options\n\nRun `ssstar create --help` and `ssstar extract --help` to get the complete CLI usage documentation for archive creation\nand extraction, respectively. There are a few command line options that are particularly likely to be of interest:\n\n### Using a custom S3 endpoint\n\n`ssstar` is developed and tested against AWS S3, however it should work with any object storage system that provides an\nS3-compatible API. In particular, most of the automated tests our CI system runs actually use [Minio](https://min.io)\nand not the real S3 API. To use `ssstar` with an S3-compatible API, use the `--s3-endpoint` option. For example, if\nyou have a Minio server running at `127.0.7.1:30000`, using default `minioadmin` credentials, you can use it with\n`ssstar` like this:\n\n```shell\nssstar --s3-endpoint http://127.0.0.1:30000 \\\n  --aws-access-key-id minioadmin --aws-secret-access-key minio-admin \\\n  ...\n```\n\n### Controlling Concurrency\n\nThe `--max-concurrent-requests` argument controls how many concurrent S3 API operations will be performed in each stage\nof the archive creation or extraction process. The default is 10, because that is what the AWS CLI uses. However if\nyou are running `ssstar` on an EC2 instance with multi-gigabit Ethernet connectivity to S3, 10 concurrent requests may\nnot be enough to saturate the network connection. Experiment with larger values to see if you experience faster\ntransfer times with more concurrency.\n\n## Usage (in a Rust project)\n\nThe library crate [`ssstar`](https://crates.io/crates/ssstar) is the engine that powers the `ssstar` CLI. When we wrote\n`ssstar` we deliberately kept all of the functionality in a library crate with a thin CLI wrapper on top, because\n`ssstar` is being used internally in Elastio to power our upcoming S3 backup feature. You too can integrate `ssstar`\nfunctionality into your Rust application. Just add `ssstar` as a dependency in your `Cargo.toml`:\n\n```toml\n[dependencies]\nssstar = \"0.7.3\"\n```\n\nSee the [docs.rs](https://docs.rs/ssstar) documentation for `ssstar` for more details and some examples. You can also\nlook at the `ssstar` CLI code [`ssstar-cli/main.rs`](`main.rs`) to see how we implemented our CLI in terms of the\n`ssstar` library crate.\n\n## License\n\nLicensed under either of\n\n- Apache License, Version 2.0\n  ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)\n- MIT license\n  ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)\n\nat your option.\n\n## Contribution\n\nUnless you explicitly state otherwise, any contribution intentionally submitted\nfor inclusion in the work by you, as defined in the Apache-2.0 license, shall be\ndual licensed as above, without any additional terms or conditions.\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felastio%2Fssstar","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Felastio%2Fssstar","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felastio%2Fssstar/lists"}