https://github.com/codemonauts/s3-ecouploader
Sync a folder to an S3 bucket and try to use not much ram, bandwith and storage space
https://github.com/codemonauts/s3-ecouploader
hacktoberfest
Last synced: 5 months ago
JSON representation
Sync a folder to an S3 bucket and try to use not much ram, bandwith and storage space
- Host: GitHub
- URL: https://github.com/codemonauts/s3-ecouploader
- Owner: codemonauts
- Created: 2020-01-21T19:31:42.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2020-02-12T10:48:27.000Z (about 6 years ago)
- Last Synced: 2025-04-09T14:12:53.641Z (11 months ago)
- Topics: hacktoberfest
- Language: Go
- Homepage:
- Size: 12.7 KB
- Stars: 4
- Watchers: 3
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# s3-ecouploader
This tool was created to run on an embedded low-end NAS of one of our customers which was running some kind of very limited
Linux so we decided to use Go and compile a static binary. To also save traffic and stoarge costs, this tool first
checks if a file exists in the destination bucket. If so, it also checks the `ETag` provided by S3 (which is just the
MD5 hash of this file). This way we only upload a file if it is not present in the bucket or changed since the last
upload.
## Credentials
The tool will read it's credentials from the default awscli config file located at `~/.aws/credentials`.
## Usage
```
-bucket string
Destination S3 Bucket
-region string
Region of the S3 Bucket
-src string
Local folder to backup
-dest string
Remote prefix for S3
-debug
Enable debug logging
-force
Skip hashing and upload all files
-stdin
Read a list of files from stdin
```
## Optimize runtime for very big folders
When running this tool every day on a big folder, most of the files didn't got touched since last run and therefore
don't need to be processed by this tool. One can now use `find` to get a list of files which got modified a short while
ago. This way the amount of files which have to be processed by this tool drops significantly. We run our backup
scripts every night and start the script like this:
```
find /mnt/data -mtime -2 -type f | ./s3-ecouploader -stdin ....
```
This command will process all files (`-type f`) which have a modification timestamp (`mtime`) between now and two days ago
(`-2`).