{"id":13437408,"url":"https://github.com/whitfin/s3-meta","last_synced_at":"2025-04-19T11:57:44.869Z","repository":{"id":57666134,"uuid":"143210774","full_name":"whitfin/s3-meta","owner":"whitfin","description":"Gather metadata about your S3 buckets","archived":false,"fork":false,"pushed_at":"2021-01-30T23:21:11.000Z","size":49,"stargazers_count":49,"open_issues_count":1,"forks_count":6,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-29T07:33:55.053Z","etag":null,"topics":["aws","aws-s3","metadata","tooling"],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/whitfin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-08-01T21:32:55.000Z","updated_at":"2025-02-02T05:05:40.000Z","dependencies_parsed_at":"2022-09-26T20:31:39.340Z","dependency_job_id":null,"html_url":"https://github.com/whitfin/s3-meta","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whitfin%2Fs3-meta","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whitfin%2Fs3-meta/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whitfin%2Fs3-meta/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whitfin%2Fs3-meta/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/whitfin","download_url":"https://codeload.github.com/whitfin/s3-meta/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248933341,"owners_count":21185460,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-s3","metadata","tooling"],"created_at":"2024-07-31T03:00:56.729Z","updated_at":"2025-04-19T11:57:44.850Z","avatar_url":"https://github.com/whitfin.png","language":"Rust","funding_links":[],"categories":["Applications","应用","应用 Applications"],"sub_categories":["Utilities","实用","公用事业 Utilities"],"readme":"# S3 Meta\n[![Crates.io](https://img.shields.io/crates/v/s3-meta.svg)](https://crates.io/crates/s3-meta)\n[![Build Status](https://img.shields.io/github/workflow/status/whitfin/s3-meta/CI)](https://github.com/whitfin/s3-meta/actions)\n\n**This tool has been migrated into [s3-utils](https://github.com/whitfin/s3-utils), please use that crate for future updates.**\n\nA simple tool to gather data about an S3 bucket (or subsection thereof). Designed to be simple, and (sort-of) shell consumable. Metadata will be added as it becomes available, and S3 interaction is controlled by [rusoto_s3](https://crates.io/crates/rusoto_s3).\n\n## Installation\n\nYou can install `s3-meta` from either this repository, or from Crates (once it's published):\n\n```shell\n# install from Cargo\n$ cargo install s3-meta\n\n# install the latest from GitHub\n$ cargo install --git https://github.com/whitfin/s3-meta.git\n```\n\n## Usage\n\nCredentials can be configured by following the instructions on the [AWS Documentation](https://docs.aws.amazon.com/cli/latest/userguide/cli-environment.html), although examples will use environment variables for the sake of clarity.\n\nYou can retrieve metadata about a bucket using the following form (making sure your region matches your bucket AWS region):\n\n```shell\n$ AWS_ACCESS_KEY_ID=MY_ACCESS_KEY_ID \\\n    AWS_SECRET_ACCESS_KEY=MY_SECRET_ACCESS_KEY \\\n    AWS_DEFAULT_REGION=us-west-2 \\\n    s3-meta my.bucket.name\n```\n\nIf you want to retrieve metadata about a \"subdirectory\" of a bucket, you can provide a prefix using this format:\n\n```shell\n$ AWS_ACCESS_KEY_ID=MY_ACCESS_KEY_ID \\\n    AWS_SECRET_ACCESS_KEY=MY_SECRET_ACCESS_KEY \\\n    AWS_DEFAULT_REGION=us-west-2 \\\n    s3-meta my.bucket.name/my/directory/path\n```\n\nDon't forget to add a space to the start of your command if you're going to inline your credentials as above!\n\n## Output\n\nOutput is pretty straightforward, and follows a relatively simple format which is easily extensible, and hopefully convenient in shell pipelines. There may be changes made to this format to make it easier to consume (spaces placed to make splitting easier, unformatted numbers, etc).\n\nBelow is an example based on a real S3 bucket (although the file names have been tweaked :)):\n\n```\n[general]\ntotal_time=7s\ntotal_space=1.94TB\ntotal_files=51,152\n\n[file_size]\naverage_file_size=37.95MB\naverage_file_bytes=37949529\nlargest_file_size=1.82GB\nlargest_file_bytes=1818900684\nlargest_file_name=path/to/my_largest_file.txt.gz\nsmallest_file_size=54B\nsmallest_file_bytes=54\nsmallest_file_name=path/to/my_smallest_file.txt.gz\nsmallest_file_others=12\n\n[extensions]\nunique_extensions=1\nmost_frequent_extension=gz\n\n[modification]\nearliest_file_date=2016-06-11T17:36:57.000Z\nearliest_file_name=path/to/my_earliest_file.txt.gz\nearliest_file_others=3\nlatest_file_date=2017-01-01T00:03:19.000Z\nlatest_file_name=path/to/my_latest_file.txt.gz\n```\n\nThis sample is based on the initial builds of `s3-meta`. Depending on when you come to this tool, there may be more (or less) included in the output above.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwhitfin%2Fs3-meta","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwhitfin%2Fs3-meta","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwhitfin%2Fs3-meta/lists"}