{"id":22051594,"url":"https://github.com/parlaynu/s3blast","last_synced_at":"2026-04-29T06:38:12.313Z","repository":{"id":175449466,"uuid":"653863352","full_name":"parlaynu/s3blast","owner":"parlaynu","description":"Multi-threaded uploading to an s3 or r2 bucket. Single file or full directory hierarchy.","archived":false,"fork":false,"pushed_at":"2023-11-30T20:38:02.000Z","size":26,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-23T15:34:09.562Z","etag":null,"topics":["aws-s3","cloudflare-r2","uploader"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/parlaynu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-14T22:56:04.000Z","updated_at":"2023-11-30T20:25:12.000Z","dependencies_parsed_at":"2023-11-30T21:40:12.605Z","dependency_job_id":null,"html_url":"https://github.com/parlaynu/s3blast","commit_stats":null,"previous_names":["studio1767/s3blast","parlaynu/s3blast"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/parlaynu/s3blast","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parlaynu%2Fs3blast","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parlaynu%2Fs3blast/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parlaynu%2Fs3blast/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parlaynu%2Fs3blast/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/parlaynu","download_url":"https://codeload.github.com/parlaynu/s3blast/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parlaynu%2Fs3blast/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262352562,"owners_count":23297684,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-s3","cloudflare-r2","uploader"],"created_at":"2024-11-30T15:09:53.505Z","updated_at":"2026-04-29T06:38:07.286Z","avatar_url":"https://github.com/parlaynu.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AWS S3 and Cloudflare R2 Uploader\n\nA fairly simple application that uploads files from local storage to an S3 or R2 bucket. It scans a source\ndirectory recursively looking for files, and uploads whatever is found.\n\nIt does two things to get uploads running in parallel:\n\n* uploads multiple files at the same time (controlled by the number of goroutines)\n* uses the s3 manager uploader so individual files upload in parallel if the file is large enough\n\nResults from a very simple test, uploading 22MBytes in 150 files using a home internet connection:\n\n| Goroutines | Rate         |\n|------------|--------------|\n| 1          | 653 kB/s     |\n| 2          | 1.3 MB/s     |\n| 3          | 1.7 MB/s     |\n| 4          | 2.0 MB/s     |\n| 5          | 2.1 MB/s     |\n| 10         | 2.1 MB/s     |\n\nHow many goroutines will be optimal for you will vary depending on the file sizes and your internet connection\nupload bandwidth. I noticed the upload speed decreasing with very large numbers of goroutines, so don't just set \nit at a ridiculous number and expect to get great results.\n\nIn one of my use cases, the uploads run 20% faster than using the aws cli. Your mileage will vary.\n\nTo run this tool, you will need:\n\n* an AWS account\n* an s3 bucket\n* an AWS profile with permissions to upload to the bucket\n\nOr:\n\n* a Cloudflare account\n* an r3 bucket\n* permissions to upload to the bucket\n\n\nIf you can upload to the bucket using the AWS CLI, then this will work for you as well.\n\n## Operation\n\nThis tool is designed so it can be run multiple times over the same source and bucket/key and\nonly new or modified files will be uploaded. \n\nNote: it never deletes files in the bucket. \n\nFile contents are tracked using sha256 hashes. This hash of the file content is calculated during scanning\nof the source and is also stored in the object in S3 as custom metadata when it is uploaded.\n\nFiles are only uploaded if they don't already exist in S3, or if the local and S3 content hashes\ndon't match.\n\nAdditionally, the exit status is zero if there were no failures, and non-zero if there were any failures. \nYou could use this, for example, in a bash script to keep repeating the upload until it completes \nwithout failure.\n\n## Usage\n\nThe usage is:\n\n    Usage: s3blast [options] \u003cfiles\u003e \u003cs3url\u003e\n      -c int\n        \tmax number of files to upload (default -1)\n      -d\tignore dot-files and dot-directories\n      -n int\n        \tthe number of upload workers (default 2)\n      -p string\n        \taws s3 credentials profile (default \"default\")\n      -r\tuse reduced redundancy storage class\n      -v\tverbose progress reporting\n\nThe file root can be a directory in which case it is traversed recursively looking for files. Or, it can be\na single file.\n\nAn example:\n\n    s3blast -p myprofile -n 4 ~/data s3://mybucket/my/key/prefix/\n\nThis will recursively search the directory `~/data` looking for files to upload. \n\nIn this example, the S3 key will be the prefix `my/key/prefix/` provided on the commandline joined with the relative path \nof the file under the `~/data` directory.\n\nAn equivalent upload command using the aws cli looks like this:\n\n    aws --profile myprofile s3 cp --recursive ~/data s3://mybucket/my/key/prefix\n\n### R2 Profiles\n\nA profile for using a R2 bucket is more involved than for an S3 bucket as you need to specify a `url_endpoint`.\n\nThe credentials file entry is the same as for S3, for example:\n\n    [my_r2_bucket]\n    aws_access_key_id = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n    aws_secret_access_key = yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy\n\nThe config entry is where you need to specify the `url_endpoint` and looks like this:\n\n    [profile my_r2_bucket]\n    output = json\n    region = auto\n    services = my_r2_bucket\n\n    [services my_r2_bucket]\n    s3 =\n      endpoint_url = https://zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz.r2.cloudflarestorage.com\n\nThe credentials and account Id needed are all available from your Cloudflare console. \nDocumentation is [here](https://developers.cloudflare.com/r2/).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparlaynu%2Fs3blast","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fparlaynu%2Fs3blast","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparlaynu%2Fs3blast/lists"}