{"id":13542348,"url":"https://github.com/ankane/s3tk","last_synced_at":"2025-11-17T14:19:53.124Z","repository":{"id":57463488,"uuid":"103362106","full_name":"ankane/s3tk","owner":"ankane","description":"A security toolkit for Amazon S3","archived":false,"fork":false,"pushed_at":"2024-10-15T02:54:21.000Z","size":110,"stargazers_count":458,"open_issues_count":1,"forks_count":26,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-04-14T10:42:31.717Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ankane.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-09-13T06:28:11.000Z","updated_at":"2025-03-28T16:06:35.000Z","dependencies_parsed_at":"2024-01-13T19:49:55.597Z","dependency_job_id":"f0a9a00b-d0c7-478b-80e8-59bb65e16471","html_url":"https://github.com/ankane/s3tk","commit_stats":{"total_commits":201,"total_committers":1,"mean_commits":201.0,"dds":0.0,"last_synced_commit":"34bcf9dafcddebe4c9bc9236774623fbe17d3b21"},"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fs3tk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fs3tk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fs3tk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ankane%2Fs3tk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ankane","download_url":"https://codeload.github.com/ankane/s3tk/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254374536,"owners_count":22060612,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T10:01:05.242Z","updated_at":"2025-11-17T14:19:53.118Z","avatar_url":"https://github.com/ankane.png","language":"Python","readme":"# s3tk\n\nA security toolkit for Amazon S3\n\n![Screenshot](https://gist.githubusercontent.com/ankane/13a9230353c78c0d5c35fd9319a23d98/raw/434b9c54bff9d41c398aa3b57f0d0494217ef7fa/console2.gif)\n\n:tangerine: Battle-tested at [Instacart](https://www.instacart.com/opensource)\n\n## Installation\n\nRun:\n\n```sh\npip install s3tk\n```\n\nYou can use the [AWS CLI](https://github.com/aws/aws-cli) or [AWS Vault](https://github.com/99designs/aws-vault) to set up your AWS credentials:\n\n```sh\npip install awscli\naws configure\n```\n\nSee [IAM policies](#iam-policies) needed for each command.\n\n## Commands\n\n### Scan\n\nScan your buckets for:\n\n- ACL open to public\n- policy open to public\n- public access blocked\n- logging enabled\n- versioning enabled\n- default encryption enabled\n\n```sh\ns3tk scan\n```\n\nOnly run on specific buckets\n\n```sh\ns3tk scan my-bucket my-bucket-2\n```\n\nAlso works with wildcards\n\n```sh\ns3tk scan \"my-bucket*\"\n```\n\nConfirm correct log bucket(s) and prefix\n\n```\ns3tk scan --log-bucket my-s3-logs --log-bucket other-region-logs --log-prefix \"{bucket}/\"\n```\n\nCheck CloudTrail object-level logging [experimental]\n\n```sh\ns3tk scan --object-level-logging\n```\n\nSkip logging, versioning, or default encryption\n\n```sh\ns3tk scan --skip-logging --skip-versioning --skip-default-encryption\n```\n\nGet email notifications of failures (via SNS)\n\n```sh\ns3tk scan --sns-topic arn:aws:sns:...\n```\n\n### List Policy\n\nList bucket policies\n\n```sh\ns3tk list-policy\n```\n\nOnly run on specific buckets\n\n```sh\ns3tk list-policy my-bucket my-bucket-2\n```\n\nShow named statements\n\n```sh\ns3tk list-policy --named\n```\n\n### Set Policy\n\n**Note:** This replaces the previous policy\n\nOnly private uploads\n\n```sh\ns3tk set-policy my-bucket --no-object-acl\n```\n\n### Delete Policy\n\nDelete policy\n\n```sh\ns3tk delete-policy my-bucket\n```\n\n### Block Public Access\n\nBlock public access on specific buckets\n\n```sh\ns3tk block-public-access my-bucket my-bucket-2\n```\n\nUse the `--dry-run` flag to test\n\n### Enable Logging\n\nEnable logging on all buckets\n\n```sh\ns3tk enable-logging --log-bucket my-s3-logs\n```\n\nOnly on specific buckets\n\n```sh\ns3tk enable-logging my-bucket my-bucket-2 --log-bucket my-s3-logs\n```\n\nSet log prefix (`{bucket}/` by default)\n\n```sh\ns3tk enable-logging --log-bucket my-s3-logs --log-prefix \"logs/{bucket}/\"\n```\n\nUse the `--dry-run` flag to test\n\nA few notes about logging:\n\n- buckets with logging already enabled are not updated at all\n- the log bucket must in the same region as the source bucket - run this command multiple times for different regions\n- it can take over an hour for logs to show up\n\n### Enable Versioning\n\nEnable versioning on all buckets\n\n```sh\ns3tk enable-versioning\n```\n\nOnly on specific buckets\n\n```sh\ns3tk enable-versioning my-bucket my-bucket-2\n```\n\nUse the `--dry-run` flag to test\n\n### Enable Default Encryption\n\nEnable [default encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html) on all buckets\n\n```sh\ns3tk enable-default-encryption\n```\n\nOnly on specific buckets\n\n```sh\ns3tk enable-default-encryption my-bucket my-bucket-2\n```\n\nThis does not encrypt existing objects - use the `encrypt` command for this\n\nUse the `--dry-run` flag to test\n\n### Scan Object ACL\n\nScan ACL on all objects in a bucket\n\n```sh\ns3tk scan-object-acl my-bucket\n```\n\nOnly certain objects\n\n```sh\ns3tk scan-object-acl my-bucket --only \"*.pdf\"\n```\n\nExcept certain objects\n\n```sh\ns3tk scan-object-acl my-bucket --except \"*.jpg\"\n```\n\n### Reset Object ACL\n\nReset ACL on all objects in a bucket\n\n```sh\ns3tk reset-object-acl my-bucket\n```\n\nThis makes all objects private. See [bucket policies](#bucket-policies) for how to enforce going forward.\n\nUse the `--dry-run` flag to test\n\nSpecify certain objects the same way as [scan-object-acl](#scan-object-acl)\n\n### Encrypt\n\nEncrypt all objects in a bucket with [server-side encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html)\n\n```sh\ns3tk encrypt my-bucket\n```\n\nUse [S3-managed keys](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html) by default. For [KMS-managed keys](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html), use:\n\n```sh\ns3tk encrypt my-bucket --kms-key-id arn:aws:kms:...\n```\n\nFor [customer-provided keys](https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html), use:\n\n```sh\ns3tk encrypt my-bucket --customer-key secret-key\n```\n\nUse the `--dry-run` flag to test\n\nSpecify certain objects the same way as [scan-object-acl](#scan-object-acl)\n\n**Note:** Objects will lose any custom ACL\n\n### Delete Unencrypted Versions\n\nDelete all unencrypted versions of objects in a bucket\n\n```sh\ns3tk delete-unencrypted-versions my-bucket\n```\n\nFor safety, this will not delete any current versions of objects\n\nUse the `--dry-run` flag to test\n\nSpecify certain objects the same way as [scan-object-acl](#scan-object-acl)\n\n### Scan DNS\n\nScan Route 53 for buckets to make sure you own them\n\n```sh\ns3tk scan-dns\n```\n\nOtherwise, you may be susceptible to [subdomain takeover](https://hackerone.com/reports/207576)\n\n## Credentials\n\nCredentials can be specified in `~/.aws/credentials` or with environment variables. See [this guide](https://boto3.readthedocs.io/en/latest/guide/configuration.html) for an explanation of environment variables.\n\nYou can specify a profile to use with:\n\n```sh\nAWS_PROFILE=your-profile s3tk\n```\n\n## IAM Policies\n\nHere are the permissions needed for each command. Only include statements you need.\n\n```json\n{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Sid\": \"Scan\",\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"s3:ListAllMyBuckets\",\n                \"s3:GetBucketAcl\",\n                \"s3:GetBucketPolicy\",\n                \"s3:GetBucketPublicAccessBlock\",\n                \"s3:GetBucketLogging\",\n                \"s3:GetBucketVersioning\",\n                \"s3:GetEncryptionConfiguration\"\n            ],\n            \"Resource\": \"*\"\n        },\n        {\n            \"Sid\": \"ScanObjectLevelLogging\",\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"cloudtrail:ListTrails\",\n                \"cloudtrail:GetTrail\",\n                \"cloudtrail:GetEventSelectors\",\n                \"s3:GetBucketLocation\"\n            ],\n            \"Resource\": \"*\"\n        },\n        {\n            \"Sid\": \"ScanDNS\",\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"s3:ListAllMyBuckets\",\n                \"route53:ListHostedZones\",\n                \"route53:ListResourceRecordSets\"\n            ],\n            \"Resource\": \"*\"\n        },\n        {\n            \"Sid\": \"ListPolicy\",\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"s3:ListAllMyBuckets\",\n                \"s3:GetBucketPolicy\"\n            ],\n            \"Resource\": \"*\"\n        },\n        {\n            \"Sid\": \"SetPolicy\",\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"s3:PutBucketPolicy\"\n            ],\n            \"Resource\": \"*\"\n        },\n        {\n            \"Sid\": \"DeletePolicy\",\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"s3:DeleteBucketPolicy\"\n            ],\n            \"Resource\": \"*\"\n        },\n        {\n            \"Sid\": \"BlockPublicAccess\",\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"s3:ListAllMyBuckets\",\n                \"s3:PutBucketPublicAccessBlock\"\n            ],\n            \"Resource\": \"*\"\n        },\n        {\n            \"Sid\": \"EnableLogging\",\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"s3:ListAllMyBuckets\",\n                \"s3:PutBucketLogging\"\n            ],\n            \"Resource\": \"*\"\n        },\n        {\n            \"Sid\": \"EnableVersioning\",\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"s3:ListAllMyBuckets\",\n                \"s3:PutBucketVersioning\"\n            ],\n            \"Resource\": \"*\"\n        },\n        {\n            \"Sid\": \"EnableDefaultEncryption\",\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"s3:ListAllMyBuckets\",\n                \"s3:PutEncryptionConfiguration\"\n            ],\n            \"Resource\": \"*\"\n        },\n        {\n            \"Sid\": \"ResetObjectAcl\",\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"s3:ListBucket\",\n                \"s3:GetObjectAcl\",\n                \"s3:PutObjectAcl\"\n            ],\n            \"Resource\": [\n                \"arn:aws:s3:::my-bucket\",\n                \"arn:aws:s3:::my-bucket/*\"\n            ]\n        },\n        {\n            \"Sid\": \"Encrypt\",\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"s3:ListBucket\",\n                \"s3:GetObject\",\n                \"s3:PutObject\"\n            ],\n            \"Resource\": [\n                \"arn:aws:s3:::my-bucket\",\n                \"arn:aws:s3:::my-bucket/*\"\n            ]\n        },\n        {\n            \"Sid\": \"DeleteUnencryptedVersions\",\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"s3:ListBucketVersions\",\n                \"s3:GetObjectVersion\",\n                \"s3:DeleteObjectVersion\"\n            ],\n            \"Resource\": [\n                \"arn:aws:s3:::my-bucket\",\n                \"arn:aws:s3:::my-bucket/*\"\n            ]\n        }\n    ]\n}\n```\n\n## Access Logs\n\n[Amazon Athena](https://aws.amazon.com/athena/) is great for querying S3 logs. Create a table (thanks to [this post](http://aws.mannem.me/?p=1462) for the table structure) with:\n\n```sql\nCREATE EXTERNAL TABLE my_bucket (\n    bucket_owner string,\n    bucket string,\n    time string,\n    remote_ip string,\n    requester string,\n    request_id string,\n    operation string,\n    key string,\n    request_verb string,\n    request_url string,\n    request_proto string,\n    status_code string,\n    error_code string,\n    bytes_sent string,\n    object_size string,\n    total_time string,\n    turn_around_time string,\n    referrer string,\n    user_agent string,\n    version_id string\n)\nROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe'\nWITH SERDEPROPERTIES (\n    'serialization.format' = '1',\n    'input.regex' = '([^ ]*) ([^ ]*) \\\\[(.*?)\\\\] ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) \\\\\\\"([^ ]*) ([^ ]*) (- |[^ ]*)\\\\\\\" (-|[0-9]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) (\\\"[^\\\"]*\\\\\") ([^ ]*)$'\n) LOCATION 's3://my-s3-logs/my-bucket/';\n```\n\nChange the last line to point to your log bucket (and prefix) and query away\n\n```sql\nSELECT\n    date_parse(time, '%d/%b/%Y:%H:%i:%S +0000') AS time,\n    request_url,\n    remote_ip,\n    user_agent\nFROM\n    my_bucket\nWHERE\n    requester = '-'\n    AND status_code LIKE '2%'\n    AND request_url LIKE '/some-keys%'\nORDER BY 1\n```\n\n## CloudTrail Logs\n\nAmazon Athena is also great for querying CloudTrail logs. Create a table (thanks to [this post](https://www.1strategy.com/blog/2017/07/25/auditing-aws-activity-with-cloudtrail-and-athena/) for the table structure) with:\n\n```sql\nCREATE EXTERNAL TABLE cloudtrail_logs (\n    eventversion STRING,\n    userIdentity STRUCT\u003c\n        type:STRING,\n        principalid:STRING,\n        arn:STRING,\n        accountid:STRING,\n        invokedby:STRING,\n        accesskeyid:STRING,\n        userName:String,\n        sessioncontext:STRUCT\u003c\n            attributes:STRUCT\u003c\n                mfaauthenticated:STRING,\n                creationdate:STRING\u003e,\n            sessionIssuer:STRUCT\u003c\n                type:STRING,\n                principalId:STRING,\n                arn:STRING,\n                accountId:STRING,\n                userName:STRING\u003e\u003e\u003e,\n    eventTime STRING,\n    eventSource STRING,\n    eventName STRING,\n    awsRegion STRING,\n    sourceIpAddress STRING,\n    userAgent STRING,\n    errorCode STRING,\n    errorMessage STRING,\n    requestId  STRING,\n    eventId  STRING,\n    resources ARRAY\u003cSTRUCT\u003c\n        ARN:STRING,\n        accountId:STRING,\n        type:STRING\u003e\u003e,\n    eventType STRING,\n    apiVersion  STRING,\n    readOnly BOOLEAN,\n    recipientAccountId STRING,\n    sharedEventID STRING,\n    vpcEndpointId STRING,\n    requestParameters STRING,\n    responseElements STRING,\n    additionalEventData STRING,\n    serviceEventDetails STRING\n)\nROW FORMAT SERDE 'com.amazon.emr.hive.serde.CloudTrailSerde'\nSTORED  AS INPUTFORMAT 'com.amazon.emr.cloudtrail.CloudTrailInputFormat'\nOUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'\nLOCATION  's3://my-cloudtrail-logs/'\n```\n\nChange the last line to point to your CloudTrail log bucket and query away\n\n```sql\nSELECT\n    eventTime,\n    eventName,\n    userIdentity.userName,\n    requestParameters\nFROM\n    cloudtrail_logs\nWHERE\n    eventName LIKE '%Bucket%'\nORDER BY 1\n```\n\n## Best Practices\n\nKeep things simple and follow the principle of least privilege to reduce the chance of mistakes.\n\n- Strictly limit who can perform bucket-related operations\n- Avoid mixing objects with different permissions in the same bucket (use a bucket policy to enforce this)\n- Don’t specify public read permissions on a bucket level (no `GetObject` in bucket policy)\n- Monitor configuration frequently for changes\n\n## Bucket Policies\n\nOnly private uploads\n\n```json\n{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Effect\": \"Deny\",\n            \"Principal\": \"*\",\n            \"Action\": \"s3:PutObjectAcl\",\n            \"Resource\": \"arn:aws:s3:::my-bucket/*\"\n        }\n    ]\n}\n```\n\n## Performance\n\nFor commands that iterate over bucket objects (`scan-object-acl`, `reset-object-acl`, `encrypt`, and `delete-unencrypted-versions`), run s3tk on an EC2 server for minimum latency.\n\n## Notes\n\nThe `set-policy`, `block-public-access`, `enable-logging`, `enable-versioning`, and `enable-default-encryption` commands are provided for convenience. We recommend [Terraform](https://www.terraform.io/) for managing your buckets.\n\n```tf\nresource \"aws_s3_bucket\" \"my_bucket\" {\n  bucket = \"my-bucket\"\n  acl    = \"private\"\n\n  logging {\n    target_bucket = \"my-s3-logs\"\n    target_prefix = \"my-bucket/\"\n  }\n\n  versioning {\n    enabled = true\n  }\n}\n\nresource \"aws_s3_bucket_public_access_block\" \"my_bucket\" {\n  bucket = \"${aws_s3_bucket.my_bucket.id}\"\n\n  block_public_acls       = true\n  block_public_policy     = true\n  ignore_public_acls      = true\n  restrict_public_buckets = true\n}\n```\n\n## Upgrading\n\nRun:\n\n```sh\npip install s3tk --upgrade\n```\n\nTo use master, run:\n\n```sh\npip install git+https://github.com/ankane/s3tk.git --upgrade\n```\n\n## Docker\n\nRun:\n\n```sh\ndocker run -it ankane/s3tk aws configure\n```\n\nCommit your credentials:\n\n```sh\ndocker commit $(docker ps -l -q) my-s3tk\n```\n\nAnd run:\n\n```sh\ndocker run -it my-s3tk s3tk scan\n```\n\n## History\n\nView the [changelog](https://github.com/ankane/s3tk/blob/master/CHANGELOG.md)\n\n## Contributing\n\nEveryone is encouraged to help improve this project. Here are a few ways you can help:\n\n- [Report bugs](https://github.com/ankane/s3tk/issues)\n- Fix bugs and [submit pull requests](https://github.com/ankane/s3tk/pulls)\n- Write, clarify, or fix documentation\n- Suggest or add new features\n\nTo get started with development:\n\n```sh\ngit clone https://github.com/ankane/s3tk.git\ncd s3tk\npip install -r requirements.txt\n```\n","funding_links":[],"categories":["Miscellaneous","Python","others","Python (1887)"],"sub_categories":["Buckets"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fankane%2Fs3tk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fankane%2Fs3tk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fankane%2Fs3tk/lists"}