{"id":37981736,"url":"https://github.com/momokatte/s3-bulk-delete","last_synced_at":"2026-01-16T18:35:20.487Z","repository":{"id":57509842,"uuid":"161553957","full_name":"momokatte/s3-bulk-delete","owner":"momokatte","description":"A Go app for bulk-deleting files from an Amazon S3 bucket using the Multi-Object Delete operation.","archived":false,"fork":false,"pushed_at":"2019-01-10T03:45:21.000Z","size":29,"stargazers_count":8,"open_issues_count":1,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-08-15T03:55:41.922Z","etag":null,"topics":["amazon-s3","amazon-s3-bucket","aws-s3","concurrent","delete-multiple","file-delete-utility","file-deletion","golang","golang-application","multithreaded","parallel","s3"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/momokatte.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-12-12T22:39:15.000Z","updated_at":"2022-01-17T06:53:16.000Z","dependencies_parsed_at":"2022-09-26T17:01:01.369Z","dependency_job_id":null,"html_url":"https://github.com/momokatte/s3-bulk-delete","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/momokatte/s3-bulk-delete","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/momokatte%2Fs3-bulk-delete","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/momokatte%2Fs3-bulk-delete/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/momokatte%2Fs3-bulk-delete/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/momokatte%2Fs3-bulk-delete/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/momokatte","download_url":"https://codeload.github.com/momokatte/s3-bulk-delete/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/momokatte%2Fs3-bulk-delete/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28480951,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T11:59:17.896Z","status":"ssl_error","status_checked_at":"2026-01-16T11:55:55.838Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon-s3","amazon-s3-bucket","aws-s3","concurrent","delete-multiple","file-delete-utility","file-deletion","golang","golang-application","multithreaded","parallel","s3"],"created_at":"2026-01-16T18:35:19.744Z","updated_at":"2026-01-16T18:35:20.478Z","avatar_url":"https://github.com/momokatte.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"s3-bulk-delete\n==============\n\nA Go app for bulk-deleting files from an Amazon S3 bucket using the Multi-Object Delete operation.\n\nSupports stop/resume: optionally writes completed batch numbers to a file, and reads from that file on startup to load batch numbers to skip.\n\n\nUsage\n-----\n\nThe application will receive a list of S3 keys via standard input, one key per line. Example input:\n\n```\nprefix_01/file_01.dat\nprefix_01/file_02.dat\nprefix_01/file_03.dat\nprefix_01/file_04.dat\nprefix_02/file_01.dat\nprefix_02/file_02.dat\nprefix_02/file_03.dat\nprefix_02/file_04.dat\n```\n\nProvide the AWS region and S3 bucket name via flags:\n\n```bash\n\u003cs3-keys.txt s3-bulk-delete -region us-east-1 -bucket my-bucket -skip deleted-batches.txt\n```\n\n\nAPI Credentials\n---------------\n\nThe application supports these AWS [credential providers](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials):\n\n- environment variables\n- shared credentials file\n- EC2 Instance Metadata service\n\n\nSpecifying keys for deletion\n----------------------------\n\nThe application does not retrieve key listings from S3; they must be provided from an external source.\n\nIf you have [Amazon S3 Inventory](https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-inventory.html) enabled for your bucket, it provides file data that can be used to generate a list of keys to delete.\n\nThe [AWS command-line interface](https://docs.aws.amazon.com/cli/latest/reference/s3/ls.html) can list all keys with a specified prefix. Here's an example that uses some additional programs to filter out folder objects and file metadata:\n\n```bash\naws --region=us-east-1 s3 ls --recursive s3://my-bucket/folderol/ | grep -v \"\\/$\" | awk '{ print $4 }' \u003es3-keys.txt\n```\n\nIf you just want to delete a few hundred or thousand keys with a particular prefix without previewing them, the AWS command-line interface provides a [recursive delete option](https://docs.aws.amazon.com/cli/latest/reference/s3/rm.html).\n\n\nRequest rate and concurrency\n----------------------------\n\nThrough trial and error I have determined that the S3 Multi-Object Delete operation is governed by the [documented rate limit](https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html) of 3500 DELETE requests per second (per bucket + prefix). It's allowed to exceed that rate for short durations, but sustained deletes of 3500 or more objects per second will eventually result in throttled API responses.\n\nI've also found that API response times degrade with high concurrency and will sometimes result in \"internal error\" responses. 12 concurrent requests provides a good balance between response time and total batch throughput.\n\nWith the default settings, the application will realistically delete 3000 objects per second or just under 11 million objects per hour. If you have hundreds of millions of objects to delete under different prefixes, it would be best to split up the input keys and run multiple instances of the application.\n\n\nAPI request costs\n-----------------\n\nDELETE requests are free for 'S3 Standard' and 'S3 Intelligent-Tiering' storage classes. Pricing for other storage classes is [here](https://aws.amazon.com/s3/pricing/#Request_pricing).\n\n\nRoadmap\n-------\n\nJanuary 2019:\n\n- IAM role support\n- Dockerize and build to DockerHub\n- More metrics\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmomokatte%2Fs3-bulk-delete","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmomokatte%2Fs3-bulk-delete","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmomokatte%2Fs3-bulk-delete/lists"}