{"id":25619516,"url":"https://github.com/silverstripe-platform/s3sync","last_synced_at":"2026-05-25T14:03:35.744Z","repository":{"id":57548845,"uuid":"87475585","full_name":"silverstripe-platform/s3sync","owner":"silverstripe-platform","description":"A faster s3 backup sync from local to remote s3 bucket","archived":false,"fork":false,"pushed_at":"2018-07-11T01:35:38.000Z","size":6193,"stargazers_count":6,"open_issues_count":7,"forks_count":1,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-03-27T14:52:31.437Z","etag":null,"topics":["aws","aws-s3","golang"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/silverstripe-platform.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-04-06T21:13:38.000Z","updated_at":"2025-02-13T00:44:30.000Z","dependencies_parsed_at":"2022-08-28T11:31:26.646Z","dependency_job_id":null,"html_url":"https://github.com/silverstripe-platform/s3sync","commit_stats":null,"previous_names":["silverstripe-platform/s3sync","silverstripeltd/s3sync"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/silverstripe-platform%2Fs3sync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/silverstripe-platform%2Fs3sync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/silverstripe-platform%2Fs3sync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/silverstripe-platform%2Fs3sync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/silverstripe-platform","download_url":"https://codeload.github.com/silverstripe-platform/s3sync/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248804486,"owners_count":21164125,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-s3","golang"],"created_at":"2025-02-22T06:17:50.658Z","updated_at":"2026-05-25T14:03:28.819Z","avatar_url":"https://github.com/silverstripe-platform.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# s3sync\n\n[![CircleCI](https://circleci.com/gh/silverstripeltd/s3sync/tree/master.svg?style=svg)](https://circleci.com/gh/silverstripeltd/s3sync/tree/master)\n\ns3sync syncs files from a local directory to a AWS S3 bucket faster than the aws cli tool. It does this by being very specific in what IO operations it does. This can make a difference when there are 10 000 files and only a few files should be synced. It also uploads files concurrently.  \n \n## Installation\n\nEither download binaries for your platform from [releases](https://github.com/silverstripeltd/s3sync/releases) or use the go installation method:\n\n```bash\n$ go get github.com/silverstripeltd/s3sync/releases\n```\n\n## Usage\n\n\n```\ns3sync [options] source_directory s3://bucket_name/prefix\n\n  -debug\n    \tTurn on debug logging.\n  -dryrun\n    \tDisplays the operations that would be performed using the specified command without actually running them.\n  -exclude value\n    \tExclude all files or objects from the command that matches the specified pattern, only supports '*' \"globbing\".\n  -only-show-errors\n    \tOnly errors and warnings are displayed. All other output is suppressed.\n  -profile string\n    \tUse a specific profile from your credential file.\n  -region string\n    \tThe region to use. Overrides config/env settings.\n```\n\n## Example benchmark\n \nThis benchmark was recorded on an AWS EC2 t2.nano instance with ~25 000 files where all but two files was sup to date.\n\n### S3sync 3.5 seconds\n```bash\n$ time s3sync /var/www/ s3://sync_bucket/www --exclude *.bak\nupload: folder/file_a to s3://sync_bucket/www/folder/file_a\nupload: file_b to s3://sync_bucket/www/file_b\n\nreal    0m3.520s\nuser    0m1.068s\nsys    0m0.100s\n```\n\n### AWS CLI tool\n\n```bash 36.3 seconds\n$ time aws s3 sync /var/www/ s3://sync_bucket/www --exclude *.bak\nupload: folder/file_a to s3://sync_bucket/www/folder/file_a\nupload: file_b to s3://sync_bucket/www/file_b\n\nreal    0m36.328s\nuser    0m16.784s\nsys    0m1.664s\n```\n\n\n## development\n\n### Updating vendor libraries\n\nFirst ensure that you have installed [glide](https://glide.sh/), a dependency and vendor manager for go.\n\nAfter you have glide installed, run `glide update`. For more information on how to use glide for adding \ndependencies, see the [glide documentation](https://glide.readthedocs.io/en/latest/).\n \nUse the Makefile to run common operations\n\n - `make dev` run go fmt and go vet  \n - `make install` runs go install\n - `make release` creates binaries for linux, mac os x and windows\n - `make clean` - removes all release binaries\n - `make test` - runs all the test and runs linters and other tools, see Makefile for necessary pre-requisites.\n - `make cover` - generats test coverage and opens the result in a browser \t\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsilverstripe-platform%2Fs3sync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsilverstripe-platform%2Fs3sync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsilverstripe-platform%2Fs3sync/lists"}