Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/whitfin/s3-meta
Gather metadata about your S3 buckets
https://github.com/whitfin/s3-meta
aws aws-s3 metadata tooling
Last synced: about 1 month ago
JSON representation
Gather metadata about your S3 buckets
- Host: GitHub
- URL: https://github.com/whitfin/s3-meta
- Owner: whitfin
- License: mit
- Created: 2018-08-01T21:32:55.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2021-01-30T23:21:11.000Z (almost 4 years ago)
- Last Synced: 2024-10-28T06:58:58.649Z (about 2 months ago)
- Topics: aws, aws-s3, metadata, tooling
- Language: Rust
- Size: 47.9 KB
- Stars: 48
- Watchers: 3
- Forks: 8
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-rust - whitfin/s3-meta
- awesome-rust-zh - whitfin/s3-meta - 用于收集，有关 Amazon S3 存储桶的元数据的命令行工具。 (应用 / 实用)
- awesome-rust - whitfin/s3-meta
- awesome-rust-cn - whitfin/s3-meta
README
# S3 Meta
[](https://crates.io/crates/s3-meta)
[](https://github.com/whitfin/s3-meta/actions)
**This tool has been migrated into [s3-utils](https://github.com/whitfin/s3-utils), please use that crate for future updates.**
A simple tool to gather data about an S3 bucket (or subsection thereof). Designed to be simple, and (sort-of) shell consumable. Metadata will be added as it becomes available, and S3 interaction is controlled by [rusoto_s3](https://crates.io/crates/rusoto_s3).
## Installation
You can install `s3-meta` from either this repository, or from Crates (once it's published):
```shell
# install from Cargo
$ cargo install s3-meta
# install the latest from GitHub
$ cargo install --git https://github.com/whitfin/s3-meta.git
```
## Usage
Credentials can be configured by following the instructions on the [AWS Documentation](https://docs.aws.amazon.com/cli/latest/userguide/cli-environment.html), although examples will use environment variables for the sake of clarity.
You can retrieve metadata about a bucket using the following form (making sure your region matches your bucket AWS region):
```shell
$ AWS_ACCESS_KEY_ID=MY_ACCESS_KEY_ID \
AWS_SECRET_ACCESS_KEY=MY_SECRET_ACCESS_KEY \
AWS_DEFAULT_REGION=us-west-2 \
s3-meta my.bucket.name
```
If you want to retrieve metadata about a "subdirectory" of a bucket, you can provide a prefix using this format:
```shell
$ AWS_ACCESS_KEY_ID=MY_ACCESS_KEY_ID \
AWS_SECRET_ACCESS_KEY=MY_SECRET_ACCESS_KEY \
AWS_DEFAULT_REGION=us-west-2 \
s3-meta my.bucket.name/my/directory/path
```
Don't forget to add a space to the start of your command if you're going to inline your credentials as above!
## Output
Output is pretty straightforward, and follows a relatively simple format which is easily extensible, and hopefully convenient in shell pipelines. There may be changes made to this format to make it easier to consume (spaces placed to make splitting easier, unformatted numbers, etc).
Below is an example based on a real S3 bucket (although the file names have been tweaked :)):
```
[general]
total_time=7s
total_space=1.94TB
total_files=51,152
[file_size]
average_file_size=37.95MB
average_file_bytes=37949529
largest_file_size=1.82GB
largest_file_bytes=1818900684
largest_file_name=path/to/my_largest_file.txt.gz
smallest_file_size=54B
smallest_file_bytes=54
smallest_file_name=path/to/my_smallest_file.txt.gz
smallest_file_others=12
[extensions]
unique_extensions=1
most_frequent_extension=gz
[modification]
earliest_file_date=2016-06-11T17:36:57.000Z
earliest_file_name=path/to/my_earliest_file.txt.gz
earliest_file_others=3
latest_file_date=2017-01-01T00:03:19.000Z
latest_file_name=path/to/my_latest_file.txt.gz
```
This sample is based on the initial builds of `s3-meta`. Depending on when you come to this tool, there may be more (or less) included in the output above.