{"id":15562153,"url":"https://github.com/mrkamel/s3sync","last_synced_at":"2025-10-18T07:45:05.237Z","repository":{"id":9582829,"uuid":"11499268","full_name":"mrkamel/s3sync","owner":"mrkamel","description":"Sync S3 buckets to your filesystem","archived":false,"fork":false,"pushed_at":"2015-05-20T07:57:11.000Z","size":156,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-03T14:36:28.891Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mrkamel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-07-18T09:37:50.000Z","updated_at":"2016-07-14T05:33:55.000Z","dependencies_parsed_at":"2022-07-30T01:38:01.378Z","dependency_job_id":null,"html_url":"https://github.com/mrkamel/s3sync","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrkamel%2Fs3sync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrkamel%2Fs3sync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrkamel%2Fs3sync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrkamel%2Fs3sync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mrkamel","download_url":"https://codeload.github.com/mrkamel/s3sync/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246140542,"owners_count":20729797,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-02T16:12:00.654Z","updated_at":"2025-10-18T07:45:05.181Z","avatar_url":"https://github.com/mrkamel.png","language":"Ruby","readme":"\n# s3sync\n\ns3sync is a low-memory, highly parallelizable (via prefixes) s3 client that\nsyncs a bucket to a local filesystem.\n\n## Setup\n\nYou need to have ruby installed on your system.\nFirst, install bundler:\n\n```\n$ gem install bundler\n```\n\nThen, install s3sync's dependencies:\n\n```\n$ cd /path/to/s3sync\n$ bundle\n```\n\nCreate a config file, like e.g. config.yml:\n\n```\nendpoint: s3-eu-west-1.amazonaws.com\naccess_key: YOUR ACCESS KEY\nsecret_key: YOUR SECRET KEY\n```\n\n## Usage\n\n```\n$ ./s3sync --help\nUsage: s3sync [options]\n        --config PATH\n        --bucket BUCKET\n        --prefix PREFIX\n        --path PATH\n```\n\nThe most common way to use it is:\n\n```\n$ s3sync --config config.yml --bucket BUCKET --prefix PREFIX --path /path/to/destination\n```\n\ns3sync currently is a fetch only client, such that it will fetch all files matching the\nbucket and optional prefix unless the file already exists on the local filesystem having\nthe same file size as on s3.\n\n# Example\n\nThe strength of s3sync is that it works great when parallelized by starting it multiple\ntimes (e.g. via threads) with disjoint prefixes:\n\n```\n$ s3sync --config config.yml --bucket BUCKET --prefix images/1 --path /path/to/destination\n$ s3sync --config config.yml --bucket BUCKET --prefix images/2 --path /path/to/destination\n$ s3sync --config config.yml --bucket BUCKET --prefix images/3 --path /path/to/destination\n...\n```\n\nTo run it in parallel via your own ruby scripts, you can use something similar to:\n\n```ruby\nrequire \"thread\"\n\ndef in_parallel(collection, n)\n  queue = Queue.new\n\n  collection.each { |element| queue.push element }\n\n  threads = []\n\n  n.times do\n    threads \u003c\u003c Thread.new do\n      begin\n        until queue.empty?\n          yield queue.pop(true)\n        end\n      rescue ThreadError =\u003e e\n        # Queue empty\n      end\n    end\n  end\n\n  threads.each \u0026:join\nend\n\nprefixes = (1 .. 9).to_a\n\nin_parallel prefixes, 5 do |prefix|\n  system \"/path/to/s3sync\", \"--config\", \"/path/to/config.yml\", \"--bucket\", \"BUCKET\", \"--prefix\", \"images/#{prefix}\", \"--path\", \"/path/to/destination\"\nend\n```\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrkamel%2Fs3sync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmrkamel%2Fs3sync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrkamel%2Fs3sync/lists"}