{"id":16151290,"url":"https://github.com/stefansundin/aws","last_synced_at":"2025-03-18T19:31:04.966Z","repository":{"id":148602380,"uuid":"63985529","full_name":"stefansundin/aws","owner":"stefansundin","description":"AWS tools and snippets","archived":false,"fork":false,"pushed_at":"2022-12-30T20:14:29.000Z","size":80,"stargazers_count":14,"open_issues_count":0,"forks_count":3,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-16T23:11:55.677Z","etag":null,"topics":["aws"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stefansundin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-07-22T21:52:42.000Z","updated_at":"2022-02-12T00:32:57.000Z","dependencies_parsed_at":"2023-05-20T16:31:26.001Z","dependency_job_id":null,"html_url":"https://github.com/stefansundin/aws","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefansundin%2Faws","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefansundin%2Faws/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefansundin%2Faws/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefansundin%2Faws/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stefansundin","download_url":"https://codeload.github.com/stefansundin/aws/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244288500,"owners_count":20429018,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws"],"created_at":"2024-10-10T00:55:39.500Z","updated_at":"2025-03-18T19:31:04.951Z","avatar_url":"https://github.com/stefansundin.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Get key identity:\n```\nAWS_ACCESS_KEY_ID=AKIA.. AWS_SECRET_ACCESS_KEY=... aws sts get-caller-identity\n```\n\n## S3\n\nFind and abort stale S3 multipart uploads:\n```\nfor bucket in $(aws s3api list-buckets --query Buckets[*].Name --output text); do\n  echo \"$bucket\"\n  aws s3api list-multipart-uploads --bucket \"$bucket\" --query Uploads[*].[Key,UploadId,Initiated,Initiator.DisplayName] --output text | while read key id date user; do\n    [[ \"$key\" == \"None\" ]] \u0026\u0026 continue\n    echo \"Press enter to abort s3://$bucket/$key (initiated $date by $user)\"\n    read \u003c /dev/tty\n    aws s3api abort-multipart-upload --bucket \"$bucket\" --key \"$key\" --upload-id \"$id\"\n  done\ndone\n```\n\n### Bucket policies\n\nEven if you lock out the root user with a bucket policy, it is still able to edit/delete the bucket policy via the management console or aws cli.\n\n- https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html\n- https://aws.amazon.com/blogs/security/how-to-restrict-amazon-s3-bucket-access-to-a-specific-iam-role/\n\nGet role id with:\n```\naws iam get-role --role-name ROLE_NAME\n```\n\n```\n{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Effect\": \"Deny\",\n            \"Principal\": \"*\",\n            \"Action\": \"s3:*\",\n            \"Resource\": [\n                \"arn:aws:s3:::bucketname\",\n                \"arn:aws:s3:::bucketname/*\"\n            ],\n            \"Condition\": {\n                \"StringNotLike\": {\n                    \"aws:userId\": [\n                        \"123456789012\",\n                        \"AROAEXAMPLEID:*\"\n                    ]\n                }\n            }\n        }\n    ]\n}\n```\n\nDeny access to dangerous things:\n```\n        {\n            \"Effect\": \"Deny\",\n            \"Principal\": \"*\",\n            \"Action\": [\n                \"s3:DeleteBucket\",\n                \"s3:DeleteBucketPolicy\",\n                \"s3:DeleteBucketWebsite\",\n                \"s3:PutBucketAcl\",\n                \"s3:PutBucketCORS\",\n                \"s3:PutBucketObjectLockConfiguration\",\n                \"s3:PutBucketPolicy\",\n                \"s3:PutBucketPublicAccessBlock\",\n                \"s3:PutBucketWebsite\",\n                \"s3:PutReplicationConfiguration\"\n            ],\n            \"Resource\": \"arn:aws:s3:::bucketname\"\n        },\n        {\n            \"Effect\": \"Deny\",\n            \"Principal\": \"*\",\n            \"Action\": [\n                \"s3:PutAccelerateConfiguration\",\n                \"s3:PutAnalyticsConfiguration\",\n                \"s3:PutBucketLogging\",\n                \"s3:PutBucketNotification\",\n                \"s3:PutBucketRequestPayment\",\n                \"s3:PutBucketVersioning\",\n                \"s3:PutEncryptionConfiguration\",\n                \"s3:PutInventoryConfiguration\",\n                \"s3:PutLifecycleConfiguration\",\n                \"s3:PutMetricsConfiguration\"\n            ],\n            \"Resource\": \"arn:aws:s3:::bucketname\",\n            \"Condition\": {\n                \"StringNotLike\": {\n                    \"aws:userId\": \"123456789012\"\n                }\n            }\n        },\n        {\n            \"Effect\": \"Deny\",\n            \"Principal\": \"*\",\n            \"Action\": [\n                \"s3:BypassGovernanceRetention\",\n                \"s3:DeleteObject\",\n                \"s3:DeleteObjectVersion\",\n                \"s3:PutObjectAcl\",\n                \"s3:PutObjectLegalHold\",\n                \"s3:PutObjectRetention\",\n                \"s3:PutObjectVersionAcl\"\n            ],\n            \"Resource\": \"arn:aws:s3:::bucketname/*\",\n            \"Condition\": {\n                \"StringNotLike\": {\n                    \"aws:userId\": \"123456789012\"\n                }\n            }\n        },\n        {\n            \"Effect\": \"Deny\",\n            \"Principal\": \"*\",\n            \"Action\": [\n                \"s3:*\"\n            ],\n            \"Resource\": \"arn:aws:s3:::bucketname/*\",\n            \"Condition\": {\n                \"StringEquals\": {\n                    \"s3:object-lock-mode\": \"COMPLIANCE\"\n                }\n            }\n        }\n```\n\nVPC Endpoints for S3 are not used for cross-region requests, so `aws:sourceVpce` cannot be used in that case. Similarly, it appears that `aws:sourceVpc` does not work cross-region.\n\n### Object Lock\n\n`--object-lock-retain-until-date` is given in this format: `2019-01-01T12:00:00.000Z`\n\nCalculate Content-MD5:\n```\nruby -rbase64 -rdigest -e 'puts Base64.strict_encode64(Digest::MD5.digest(File.read(\"file.zip\")))'\n```\n\n### MFA Delete\n\nEnabling [MFA Delete](https://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html#MultiFactorAuthenticationDelete) must be done by the root user and with the aws cli. U2F is not supported.\n\nLog in with the root user and get the MFA serial number from https://console.aws.amazon.com/iam/home#/security_credentials\n\nThe MFA serial number is typically in this format:\n```\narn:aws:iam::123456789012:mfa/root-account-mfa-device\n```\n\n```\n# Enable MFA Delete:\naws s3api put-bucket-versioning --profile root --bucket bucketname --versioning-configuration Status=Enabled,MFADelete=Enabled --mfa \"mfa_serial_number mfa_code\"\n\n# Delete an object version:\naws s3api delete-object --profile root --bucket bucketname --key path/to/file --version-id longversionid --mfa \"mfa_serial_number mfa_code\"\n\n# Get versioning status:\naws s3api get-bucket-versioning --bucket bucketname\n\n# Disable MFA Delete:\naws s3api put-bucket-versioning --profile root --bucket bucketname --versioning-configuration Status=Enabled,MFADelete=Disabled --mfa \"mfa_serial_number mfa_code\"\n```\n\nIt would be great if this could be done with a U2F device. At least you can enable MFA Delete on a few buckets, and then switch back to U2F until you need to delete objects.\n\n## EC2\n\nSpecial IP addresses:\n- IPv4:\n  - 169.254.169.254 - [Instance metadata service](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html)\n    - 169.254.169.254 used to be accessible with the `instance-data` hostname, but this appears to no longer work. was this a /etc/hosts hack for amazon linux perhaps?\n  - 169.254.169.253 - [DNS](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_DHCP_Options.html#AmazonDNS)\n  - 169.254.169.123 - [Time server](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html)\n- IPv6:\n  - fd00:ec2::254\n    - Requires that the metadata option `HttpProtocolIpv6` is enabled.\n  - fd00:ec2::253\n  - fd00:ec2::123\n\nChange EBS \"Delete on Termination\" flag after launching instance:\n```\naws ec2 modify-instance-attribute --instance-id i-01234567890abcdef --block-device-mappings '[{\"DeviceName\":\"/dev/sda1\",\"Ebs\":{\"DeleteOnTermination\":false}}]'\n```\n\nUserdata environment (Ubuntu 16.04). Note that HOME is missing.\n```\n+ env\nPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\nPWD=/root\nSHLVL=1\n_=/usr/bin/env\nOLDPWD=/\n```\n\nGet instance details without using the EC2 instance metadata service:\n```\n# instance id:\ncat /sys/devices/virtual/dmi/id/board_asset_tag\n# instance type:\ncat /sys/devices/virtual/dmi/id/product_name\n```\n\n### Initialize EBS volume after restoring from snapshot\n\nhttps://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html\n\n```\nsudo dd if=/dev/xvdf of=/dev/null bs=1M\n\nsudo apt-get install -y fio\nsudo fio --filename=/dev/xvdf --rw=read --bs=128k --iodepth=32 --ioengine=libaio --direct=1 --name=volume-initialize\n```\n\nRead progress from dd with:\n```\nsudo pkill -USR1 -n -x dd\n```\n\nFaster parallel processing:\n```\nseq 0 $(($(cat /sys/block/xvdf/size) / (1 \u003c\u003c 10))) | xargs -n1 -P32 -I {} sudo dd if=/dev/xvdf of=/dev/null skip={}k count=1 bs=512 \u003e /tmp/initialize.log 2\u003e\u00261\n\nsudo fio --filename=/dev/xvdf --direct=1 --rw=randread --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=128k --rwmixread=100 --iodepth=32 --numjobs=4 --group_reporting --name=initialize\n```\n\n\n## RDS\n\nEnable binlog on RDS MySQL without having a replica:\n- Enable automated backups (1 day is fine)\n- Set `log_bin` and `binlog_format`\n- Connect and run `CALL mysql.rds_set_configuration('binlog retention hours', 4);` (168 hours is max (one week))\n\nGet default parameter groups:\n```\naws rds describe-db-cluster-parameters --db-cluster-parameter-group-name default.aurora-mysql5.7 \u003e default.aurora-mysql5.7-cluster.json\naws rds describe-db-parameters --db-parameter-group-name default.aurora-mysql5.7 \u003e default.aurora-mysql5.7-instance.json\naws rds describe-db-parameters --db-parameter-group-name default.mysql5.7 \u003e default.mysql5.7-instance.json\n\naws rds describe-engine-default-cluster-parameters --db-parameter-group-family aurora-mysql5.7 \u003e aurora-mysql5.7-cluster.json\naws rds describe-engine-default-parameters --db-parameter-group-family aurora-mysql5.7 \u003e aurora-mysql5.7-instance.json\naws rds describe-engine-default-parameters --db-parameter-group-family mysql5.7 \u003e mysql5.7.json\n```\n\nTo get the Aurora version, you have to connect with `mysql` and run `SELECT AURORA_VERSION();`.\n\n\n## Terraform\n\n```shell\nsudo apt install graphviz\nbrew install gprof2dot\nterraform graph | dot -Tpng \u003e graph.png\n```\n\n## Acronyms\n\n- PDT: GovCloud\n- LCK: Rickenbacker International Airport in Columbus, Ohio.\n- DCA: Ronald Reagan Washington National Airport in Washington, D.C.\n\n## Billing\n\nCost Explorer buckets cost and abbreviates some API operations. For example:\n- USW2-CW:MetricMonitorUsage\n- USW2-CW:GMD-Metrics\n- USW2-CW:Requests\n\nIn this case, GMD is short for GetMetricData.\n\n## Swedish region\n\n- Städer: Eskilstuna, Katrineholm, Västerås.\n- Vi har höga ambitioner och mål för den här regionen. Vi vet att det finns mycket duktigt folk att anställa här. Dessutom kommer 53 procent av energin i Sverige från förnybara källor, och det passar bra med vår ambition att ha 100 procent förnybar energi.\n- Det kommer att vara mellan 50 000 och 80 000 servrar vid varje Availabilty Zone, och det kommer att skapas många jobb. Någon exakt siffra har jag inte, men vi vet att det kommer att ge oss helt nya möjligheter till affärsutveckling och en möjlighet att skapa nya tjänster, säger Darren Mowry.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstefansundin%2Faws","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstefansundin%2Faws","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstefansundin%2Faws/lists"}