{"id":13425025,"url":"https://github.com/larrabee/s3sync","last_synced_at":"2025-04-04T16:14:59.040Z","repository":{"id":40636929,"uuid":"122167085","full_name":"larrabee/s3sync","owner":"larrabee","description":"Really fast sync tool for S3","archived":false,"fork":false,"pushed_at":"2024-01-25T13:53:01.000Z","size":12733,"stargazers_count":500,"open_issues_count":24,"forks_count":74,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-03-28T15:07:15.642Z","etag":null,"topics":["amazon","ceph-radosgw","golang","multithreading","s3","sync","sync-s3-bucket"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/larrabee.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-02-20T07:23:21.000Z","updated_at":"2025-03-18T03:05:02.000Z","dependencies_parsed_at":"2024-01-14T10:11:31.586Z","dependency_job_id":"f1ee5007-ea33-4a3d-aa83-c6333f951fbf","html_url":"https://github.com/larrabee/s3sync","commit_stats":null,"previous_names":[],"tags_count":63,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/larrabee%2Fs3sync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/larrabee%2Fs3sync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/larrabee%2Fs3sync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/larrabee%2Fs3sync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/larrabee","download_url":"https://codeload.github.com/larrabee/s3sync/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247208151,"owners_count":20901570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon","ceph-radosgw","golang","multithreading","s3","sync","sync-s3-bucket"],"created_at":"2024-07-31T00:01:02.606Z","updated_at":"2025-04-04T16:14:59.016Z","avatar_url":"https://github.com/larrabee.png","language":"Go","readme":"# S3Sync\n#### Really fast sync tool for S3\n[![Go Report Card](https://goreportcard.com/badge/github.com/larrabee/s3sync)](https://goreportcard.com/report/github.com/larrabee/s3sync) [![GoDoc](https://godoc.org/github.com/larrabee/s3sync?status.svg)](https://godoc.org/github.com/larrabee/s3sync) [![Build Status](https://travis-ci.org/larrabee/s3sync.svg?branch=master)](https://travis-ci.org/larrabee/s3sync)  \n\n## Features\n* Multi-threaded file downloading/uploading\n* Can sync to multiple ways:\n    * S3 to local FS\n    * Local FS to S3\n    * S3 to S3\n* Retrying on errors\n* Live statistics\n* Rate limiting by objects\n* Rate limiting by bandwidth\n* Flexible filters by extension, Content-Type, ETag and object mtime\n\nKey feature: very high speed.  \nAvg listing speed around 5k objects/sec for S3.  \nWith 128 workers we get avg sync speed around 2k obj/sec (small objects 1-20 kb) (limited by 1Gb uplink).  \n\n## Limitations\n* Each object is loaded into RAM. So you need `\u003cavg object size\u003e * \u003cworkers count\u003e` RAM.  \n  If you don't have enough RAM, you can use swap. A large (32-64 Gb) swap on SSD does not affect the tool performance.  \n  This happened because the tool was designed to synchronize billions of small files and optimized for this workload.\n  To avoid this you can use streaming storage drivers (now available only for S3 and FS). It's uses less RAM, but slower on small objects.\n\n## Usage\n```\n\u003e\u003e s3sync --help\nReally fast sync tool for S3\nVersionId: dev, commit: none, built at: unknown\nUsage: cli [--sn] [--sk SK] [--ss SS] [--st ST] [--sr SR] [--se SE] [--tn] [--tk TK] [--ts TS] [--tt TT] [--tr TR] [--te TE] [--s3-retry S3-RETRY] [--s3-retry-sleep S3-RETRY-SLEEP] [--s3-acl S3-ACL] [--s3-storage-class S3-STORAGE-CLASS] [--s3-keys-per-req S3-KEYS-PER-REQ] [--fs-file-perm FS-FILE-PERM] [--fs-dir-perm FS-DIR-PERM] [--fs-disable-xattr] [--fs-atomic-write] [--filter-ext FILTER-EXT] [--filter-not-ext FILTER-NOT-EXT] [--filter-ct FILTER-CT] [--filter-not-ct FILTER-NOT-CT] [--filter-after-mtime FILTER-AFTER-MTIME] [--filter-before-mtime FILTER-BEFORE-MTIME] [--filter-modified] [--filter-exist] [--filter-not-exist] [--filter-dirs] [--filter-not-dirs] [--workers WORKERS] [--debug] [--sync-log] [--sync-log-format SYNC-LOG-FORMAT] [--sync-progress] [--on-fail ON-FAIL] [--error-handling ERROR-HANDLING] [--disable-http2] [--list-buffer LIST-BUFFER] [--ratelimit-objects RATELIMIT-OBJECTS] [--ratelimit-bandwidth RATELIMIT-BANDWIDTH] SOURCE TARGET\n\nPositional arguments:\n  SOURCE\n  TARGET\n\nOptions:\n  --sn                   Don't sign request to source AWS for anonymous access\n  --sk SK                Source AWS key\n  --ss SS                Source AWS session secret\n  --st ST                Source AWS token\n  --sr SR                Source AWS Region\n  --se SE                Source AWS Endpoint\n  --tn                   Don't sign request to target AWS for anonymous access\n  --tk TK                Target AWS key\n  --ts TS                Target AWS secret\n  --tt TT                Target AWS session token\n  --tr TR                Target AWS Region\n  --te TE                Target AWS Endpoint\n  --s3-retry S3-RETRY    Max numbers of retries to sync file\n  --s3-retry-sleep S3-RETRY-SLEEP\n                         Sleep interval (sec) between sync retries on error\n  --s3-acl S3-ACL        S3 ACL for uploaded files. Possible values: private, public-read, public-read-write, aws-exec-read, authenticated-read, bucket-owner-read, bucket-owner-full-control\n  --s3-storage-class S3-STORAGE-CLASS\n                         S3 Storage Class for uploaded files.\n  --s3-keys-per-req S3-KEYS-PER-REQ\n                         Max numbers of keys retrieved via List request [default: 1000]\n  --fs-file-perm FS-FILE-PERM\n                         File permissions [default: 0644]\n  --fs-dir-perm FS-DIR-PERM\n                         Dir permissions [default: 0755]\n  --fs-disable-xattr     Disable FS xattr for storing metadata\n  --fs-atomic-write      Enable FS atomic writes. New files will be written to temp file and renamed\n  --filter-ext FILTER-EXT\n                         Sync only files with given extensions\n  --filter-not-ext FILTER-NOT-EXT\n                         Skip files with given extensions\n  --filter-ct FILTER-CT\n                         Sync only files with given Content-Type\n  --filter-not-ct FILTER-NOT-CT\n                         Skip files with given Content-Type\n  --filter-after-mtime FILTER-AFTER-MTIME\n                         Sync only files modified after given unix timestamp\n  --filter-before-mtime FILTER-BEFORE-MTIME\n                         Sync only files modified before given unix timestamp\n  --filter-modified      Sync only modified files\n  --filter-exist         Sync only files, that exist in target storage\n  --filter-not-exist     Sync only files, that doesn't exist in target storage\n  --filter-dirs          Sync only files, that ends with slash (/)\n  --filter-not-dirs      Skip files that ends with slash (/)\n  --workers WORKERS, -w WORKERS\n                         Workers count [default: 16]\n  --debug, -d            Show debug logging\n  --sync-log             Show sync log\n  --sync-log-format SYNC-LOG-FORMAT\n                         Format of sync log. Possible values: json\n  --sync-progress, -p    Show sync progress\n  --on-fail ON-FAIL, -f ON-FAIL\n                         Action on failed. Possible values: fatal, skip, skipmissing (DEPRECATED, use --error-handling instead) [default: fatal]\n  --error-handling ERROR-HANDLING\n                         Controls error handling. Sum of the values: 1 for ignoring NotFound errors, 2 for ignoring PermissionDenied errors OR 255 to ignore all errors\n  --disable-http2        Disable HTTP2 for http client\n  --list-buffer LIST-BUFFER\n                         Size of list buffer [default: 1000]\n  --ratelimit-objects RATELIMIT-OBJECTS\n                         Rate limit objects per second\n  --ratelimit-bandwidth RATELIMIT-BANDWIDTH\n                         Set bandwidth rate limit, byte/s, Allow suffixes: K, M, G\n  --help, -h             display this help and exit\n  --version              display version and exit\n```\n\nExamples:  \n* Sync Amazon S3 bucket to FS:  \n```s3sync --sk KEY --ss SECRET -w 128 s3://shared fs:///opt/backups/s3/```\n* Sync S3 bucket with custom endpoint to FS:  \n```s3sync --sk KEY --ss SECRET --se \"http://127.0.0.1:7484\" -w 128 s3://shared fs:///opt/backups/s3/```\n* Sync directory (/test) from Amazon S3 bucket to FS:  \n```s3sync --sk KEY --ss SECRET -w 128 s3://shared/test fs:///opt/backups/s3/test/```\n* Sync directory from local FS to Amazon S3:  \n```s3sync --tk KEY --ts SECRET -w 128 fs:///opt/backups/s3/ s3://shared```\n* Sync directory from local FS to Amazon S3 bucket directory:  \n```s3sync --tk KEY --ts SECRET -w 128 fs:///opt/backups/s3/test/ s3://shared/test_new/```\n* Sync one Amazon bucket to another Amazon bucket:  \n```s3sync --tk KEY2 --ts SECRET2 --sk KEY1 --ss SECRET1 -w 128 s3://shared s3://shared_new```\n* Sync S3 bucket with custom endpoint to another bucket with custom endpoint:  \n```s3sync --tk KEY2 --ts SECRET2 --sk KEY1 --ss SECRET1 --se \"http://127.0.0.1:7484\" --te \"http://127.0.0.1:7484\" -w 128 s3://shared s3://shared_new```\n* Sync one Amazon bucket directory to another Amazon bucket:  \n```s3sync --tk KEY2 --ts SECRET2 --sk KEY1 --ss SECRET1 -w 128 s3://shared/test/ s3://shared_new```\n* Use streaming S3 transfers (reduces memory, might be slower for small files):\n```s3sync --sk KEY --ss SECRET --tk KEY --ts SECRET -w 128 s3s://shared/test/ s3s://shared_new```\n\nSOURCE and TARGET should be a directory. Syncing of single file are not supported (This will not work `s3sync --sk KEY --ss SECRET s3://shared/megafile.zip fs:///opt/backups/s3/`)  \n\nYou can use filters.   \n* Timestamp filter (`--filter-after-mtime` arg) syncing only files, that has been changed after specified timestamp. Its useful for diff backups.  \n* File extension filter (`--filter-ext` arg) syncing only files, that have specified extension. Can be specified multiple times (Like this `--filter-ext .jpg --filter-ext .png --filter-ext .bmp`).\n* Content-type filter (`--filter-ct` arg) syncing only files, that have specified content-type. Can be specified multiple times.\n* Etag filter (`--filter-modified`) sync only modified files. It have few restrictions. If you are using FS storage, the files must be created using s3sync. FS storage should also support xattr.\n* There are also inverted filters (`--filter-not-ext`, `--filter-not-ct` and `--filter-before-mtime`).\n\n## Install\nDownload binary from [Release page](https://github.com/larrabee/s3sync/releases).  \nOr use docker image [larrabee/s3sync](https://hub.docker.com/repository/docker/larrabee/s3sync) like this:  \n```\ndocker run --rm -ti larrabee/s3sync --tk KEY2 --ts SECRET2 --sk KEY1 --ss SECRET1 -w 128 s3://shared/test/ s3://shared_new\n```\n\n## Building\nMinimum go version: **1.13**  \nBuild it with:\n ```\ngo mod vendor\ngo build -o s3sync ./cli \n ```\n\n## Using module\nYou can easy use s3sync in your application. See example in `cli/` folder. \n\n## License\nGPLv3\n\n## Notes\ns3sync is a non-destructive one-way sync: it does not delete files in the destination or source paths that are out of sync.\n","funding_links":[],"categories":["Go"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flarrabee%2Fs3sync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flarrabee%2Fs3sync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flarrabee%2Fs3sync/lists"}