https://github.com/silverstripe-platform/s3sync
A faster s3 backup sync from local to remote s3 bucket
https://github.com/silverstripe-platform/s3sync
aws aws-s3 golang
Last synced: about 1 month ago
JSON representation
A faster s3 backup sync from local to remote s3 bucket
- Host: GitHub
- URL: https://github.com/silverstripe-platform/s3sync
- Owner: silverstripe-platform
- Created: 2017-04-06T21:13:38.000Z (about 9 years ago)
- Default Branch: master
- Last Pushed: 2018-07-11T01:35:38.000Z (almost 8 years ago)
- Last Synced: 2025-03-27T14:52:31.437Z (over 1 year ago)
- Topics: aws, aws-s3, golang
- Language: Go
- Size: 5.91 MB
- Stars: 6
- Watchers: 13
- Forks: 1
- Open Issues: 7
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# s3sync
[](https://circleci.com/gh/silverstripeltd/s3sync/tree/master)
s3sync syncs files from a local directory to a AWS S3 bucket faster than the aws cli tool. It does this by being very specific in what IO operations it does. This can make a difference when there are 10 000 files and only a few files should be synced. It also uploads files concurrently.
## Installation
Either download binaries for your platform from [releases](https://github.com/silverstripeltd/s3sync/releases) or use the go installation method:
```bash
$ go get github.com/silverstripeltd/s3sync/releases
```
## Usage
```
s3sync [options] source_directory s3://bucket_name/prefix
-debug
Turn on debug logging.
-dryrun
Displays the operations that would be performed using the specified command without actually running them.
-exclude value
Exclude all files or objects from the command that matches the specified pattern, only supports '*' "globbing".
-only-show-errors
Only errors and warnings are displayed. All other output is suppressed.
-profile string
Use a specific profile from your credential file.
-region string
The region to use. Overrides config/env settings.
```
## Example benchmark
This benchmark was recorded on an AWS EC2 t2.nano instance with ~25 000 files where all but two files was sup to date.
### S3sync 3.5 seconds
```bash
$ time s3sync /var/www/ s3://sync_bucket/www --exclude *.bak
upload: folder/file_a to s3://sync_bucket/www/folder/file_a
upload: file_b to s3://sync_bucket/www/file_b
real 0m3.520s
user 0m1.068s
sys 0m0.100s
```
### AWS CLI tool
```bash 36.3 seconds
$ time aws s3 sync /var/www/ s3://sync_bucket/www --exclude *.bak
upload: folder/file_a to s3://sync_bucket/www/folder/file_a
upload: file_b to s3://sync_bucket/www/file_b
real 0m36.328s
user 0m16.784s
sys 0m1.664s
```
## development
### Updating vendor libraries
First ensure that you have installed [glide](https://glide.sh/), a dependency and vendor manager for go.
After you have glide installed, run `glide update`. For more information on how to use glide for adding
dependencies, see the [glide documentation](https://glide.readthedocs.io/en/latest/).
Use the Makefile to run common operations
- `make dev` run go fmt and go vet
- `make install` runs go install
- `make release` creates binaries for linux, mac os x and windows
- `make clean` - removes all release binaries
- `make test` - runs all the test and runs linters and other tools, see Makefile for necessary pre-requisites.
- `make cover` - generats test coverage and opens the result in a browser