https://github.com/orlangure/gompress

CLI tool to compress the contents of AWS S3 bucket
https://github.com/orlangure/gompress

aws go golang gzip s3

Last synced: about 2 months ago
JSON representation

CLI tool to compress the contents of AWS S3 bucket

• Host: GitHub
• URL: https://github.com/orlangure/gompress
• Owner: orlangure
• License: mit
• Created: 2020-02-08T22:35:52.000Z (over 6 years ago)
• Default Branch: master
• Last Pushed: 2020-06-23T17:31:18.000Z (about 6 years ago)
• Last Synced: 2025-03-29T09:24:06.858Z (over 1 year ago)
• Topics: aws, go, golang, gzip, s3
• Language: Go
• Size: 234 KB
• Stars: 2
• Watchers: 2
• Forks: 0
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# gompress

CLI tool to compress the contents of AWS S3 bucket.

Gompress takes files from the original location, compresses them on the fly

using gzip, and uploads to their destination, keeping the original names (only

adding `.gz`).

Gompress never keeps large files completely in memory, and never writes them to

disk. It streams data from source into destination, adding a compression layer

in the middle.

It is recommended to become familiar with [AWS S3

pricing](https://aws.amazon.com/s3/pricing/) before using this utility. Moving

files around costs money. To reduce the cost, use EC2 instances in the same

region as the buckets!

## Usage

### Running in a docker container:

```

docker run --rm \

    -e AWS_ACCESS_KEY_ID \

    -e AWS_SECRET_ACCESS_KEY \

    orlangure/gompress \

    -src-region us-east-1 \

    -src-bucket uncompressed-files \

    -src-prefix some-folder/and-sub-folder \

    -dst-region us-east-1 \

    -dst-bucket compressed-files \

    -dst-prefix a-folder \

    -keep

```

AWS credentials with sufficient permissions to list, get, create and delete

objects are required. AWS key and secret should be provided using [environment

variables](https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e---env---env-file).

When `-keep` flag is provided, original files remain in their original

location. Otherwise, when `-keep` flag is not set, original files are removed

once their compressed versions are successfully uploaded to the destination

path.

### Building from source (Go 1.13 or above is required):

```

$ git clone https://github.com/orlangure/gompress

$ cd gompress

$ go install .

$ gompress

Usage of gompress:

  -dst-bucket string

        target s3 bucket name

  -dst-prefix string

        new files will be prefixed with this value

  -dst-region string

        target region (default "us-east-1")

  -keep

        set to keep original files (remove by default)

  -src-bucket string

        source s3 bucket name

  -src-prefix string

        source file prefix

  -src-region string

        source region (default "us-east-1")

```