{"id":24366960,"url":"https://github.com/SWO-GS/retro-tag","last_synced_at":"2025-09-29T03:31:20.990Z","repository":{"id":78248470,"uuid":"197683574","full_name":"SWO-GS/retro-tag","owner":"SWO-GS","description":"Retrospectively tag AWS resources so you can work out who created them","archived":true,"fork":false,"pushed_at":"2023-12-05T17:20:26.000Z","size":83,"stargazers_count":112,"open_issues_count":5,"forks_count":11,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-01-18T12:04:16.942Z","etag":null,"topics":["aws","aws-tags","cloudtrail","tagging"],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SWO-GS.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":"audit_tag.rb","citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-07-19T01:55:43.000Z","updated_at":"2025-01-15T08:28:21.000Z","dependencies_parsed_at":"2023-12-05T18:34:06.573Z","dependency_job_id":"0926dae3-fe71-4168-a364-59807fe460f9","html_url":"https://github.com/SWO-GS/retro-tag","commit_stats":null,"previous_names":["swo-gs/retro-tag"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SWO-GS%2Fretro-tag","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SWO-GS%2Fretro-tag/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SWO-GS%2Fretro-tag/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SWO-GS%2Fretro-tag/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SWO-GS","download_url":"https://codeload.github.com/SWO-GS/retro-tag/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234584380,"owners_count":18856290,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-tags","cloudtrail","tagging"],"created_at":"2025-01-19T01:02:11.055Z","updated_at":"2025-09-29T03:31:20.678Z","avatar_url":"https://github.com/SWO-GS.png","language":"Ruby","funding_links":[],"categories":["Ruby"],"sub_categories":[],"readme":"# Retro Tag\n\n[![Software License](https://img.shields.io/github/license/gorillastack/retro-tag.svg?style=for-the-badge)](/LICENSE)\n![GitHub last commit](https://img.shields.io/github/last-commit/gorillastack/retro-tag.svg?style=for-the-badge)\n[![Powered By: GorillaStack](https://img.shields.io/badge/powered%20by-GorillaStack-green.svg?style=for-the-badge)](https://www.gorillastack.com)\n\nRetro Tag helps you retrospectively tag resources with the ARN of the user that created them and the creation date/time using the [Auto Tag](https://github.com/GorillaStack/auto-tag) engine, and supports tagging across multiple regions and across AWS accounts.\n\nThis is designed to solve the problem of [Auto Tagging](https://github.com/GorillaStack/auto-tag) existing resources in your environments. \n\n## About\n\nRetro Tag uses the log data in your AWS CloudTrail S3 bucket logs to gather information about the \"who\" and \"when\" for each of your AWS existing resources. Using this information, engineers can determine which resources are required, which are not, and can cleanup the resources, or improve their tagging.\n\n## Installation\n\nThe installation consists of a `CSV` created from AWS Athena scanning\nthe CloudTrail S3 bucket, a single `Main` CloudFormation Stack with the\nAutoTag Lambda function in the same account as the CloudTrail S3 Bucket,\nand a `Role` CloudFormation Stack deployed to each additional account where tagging\nwill be applied.\n\n### Query CloudTrail logs using AWS Athena\n\nUse AWS Athena to scan your history of CloudTrail logs in S3 and produce\na CSV file of events that we can process.\n\nWARNING: You are charged for AWS Athena based on the amount the data that is scanned.\n\n#### Table Query\n\nLogin to the AWS account and region where your CloudTrail logs S3 bucket is located and bring up the Athena service. You'll probably need to create a unique table for each AWS account in the S3 bucket.\n\nUpdate the table name, S3 bucket, S3 path including the AWS account ID to query.\n\n```sql\nCREATE EXTERNAL TABLE IF NOT EXISTS my_table_name (\n    eventversion STRING,\n    userIdentity STRUCT\u003c\n        type:STRING,\n        principalid:STRING,\n        arn:STRING,\n        accountid:STRING,\n        invokedby:STRING,\n        accesskeyid:STRING,\n        userName:STRING,\n    sessioncontext:STRUCT\u003c\n    attributes:STRUCT\u003c\n        mfaauthenticated:STRING,\n        creationdate:STRING\u003e,\n    sessionIssuer:STRUCT\u003c  \n        type:STRING,\n        principalId:STRING,\n        arn:STRING, \n        accountId:STRING,\n        userName:STRING\u003e\u003e\u003e,\n    eventTime STRING,\n    eventSource STRING,\n    eventName STRING,\n    awsRegion STRING,\n    sourceIpAddress STRING,\n    userAgent STRING,\n    errorCode STRING,\n    errorMessage STRING,\n    requestParameters STRING,\n    responseElements STRING,\n    additionalEventData STRING,\n    requestId STRING,\n    eventId STRING,\n    resources ARRAY\u003cSTRUCT\u003c\n        ARN:STRING,\n        accountId:STRING,\n        type:STRING\u003e\u003e,\n    eventType STRING,\n    apiVersion STRING,\n    readOnly STRING,\n    recipientAccountId STRING,\n    serviceEventDetails STRING,\n    sharedEventID STRING,\n    vpcEndpointId STRING\n)\nROW FORMAT SERDE 'com.amazon.emr.hive.serde.CloudTrailSerde'\nSTORED AS INPUTFORMAT 'com.amazon.emr.cloudtrail.CloudTrailInputFormat'\nOUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'\nLOCATION 's3://my-cloudtrail-bucket/dev/AWSLogs/11111111111/'\n```\n\n#### Data Query\n\nUpdate the table name, run the Athena query, and save the output to a CSV file.\n\nNOTE: You can request a longer Athena query timeout limit from AWS if the default 30 minutes is not enough. \n\n```sql\nSELECT eventTime, eventSource, eventName, awsRegion, userIdentity.accountId as \"userIdentity.accountId\", recipientAccountId, \"$path\" as key, requestParameters, responseElements\nFROM my_table_name\nWHERE\neventName in (\n    'AllocateAddress',\n    'CloneStack',\n    'CopyImage',\n    'CopySnapshot',\n    'CreateAutoScalingGroup',\n    'CreateBucket',\n    'CreateCustomerGateway',\n    'CreateDBInstance',\n    'CreateDhcpOptions',\n    'CreateFunction20150331',\n    'CreateFunction20141111',\n    'CreateImage',\n    'CreateInternetGateway',\n    'CreateLoadBalancer',\n    'CreateLogGroup',\n    'CreateNatGateway',\n    'CreateNetworkAcl',\n    'CreateNetworkInterface',\n    'CreatePipeline',\n    'CreateRole',\n    'CreateRouteTable',\n    'CreateSecurityGroup',\n    'CreateSnapshot',\n    'CreateStack',\n    'CreateSubnet',\n    'CreateTable',\n    'CreateUser',\n    'CreateVolume',\n    'CreateVpc',\n    'CreateVpnConnection',\n    'CreateVpnGateway',\n    'CreateVpcPeeringConnection',\n    'ImportSnapshot',\n    'PutMetricAlarm',\n    'PutRule',\n    'RegisterImage',\n    'RunInstances',\n    'RunJobFlow'\n)\nand eventSource in (\n    'autoscaling.amazonaws.com',\n    'datapipeline.amazonaws.com',\n    'dynamodb.amazonaws.com',\n    'ec2.amazonaws.com',\n    'events.amazonaws.com',\n    'elasticloadbalancing.amazonaws.com',\n    'elasticmapreduce.amazonaws.com',\n    'iam.amazonaws.com',\n    'lambda.amazonaws.com',\n    'logs.amazonaws.com',\n    'monitoring.amazonaws.com',\n    'opsworks.amazonaws.com',\n    'rds.amazonaws.com',\n    's3.amazonaws.com'\n)\nand errorcode is null\n```\n\n### Deploy the Main CloudFormation template\n\nIn the same account as your CloudTrail S3 bucket deploy this Main CloudFormation template in a single region. This one CloudFormation stack, in combination with the IAM Role CloudFormation stack, will have the ability to tag all regions and more than one AWS account.\n\n**Note:** An explanation of each of the parameters:\n\n```\nCloudTrailBucketName - Name of bucket where your CloudTrail logs are stored.\n\nCodeS3Bucket - Name of bucket where AutoTag code is stored. You will need to have either downloaded a zip of a AutoTag release and upload it to your own bucket, or refer to the bucket where your AutoTag code is stored if you also have AutoTag running inside the same AWS account.\n\nCodeS3Path - Path to the zip file containing the AutoTag code inside the CodeS3Bucket.\n```\n\n```bash\nexport REGION=ap-southeast-2 # set this to the region you plan to deploy to\n\nwget https://raw.githubusercontent.com/GorillaStack/retro-tag/master/cloud_formation/autotag_retro_main-template.json\n\naws cloudformation create-stack \\\n  --template-body file://autotag_retro_main-template.json \\\n  --stack-name AutoTagRetro \\\n  --parameters \\\n     ParameterKey=CloudTrailBucketName,ParameterValue=my-cloudtrail-bucket \\\n     ParameterKey=CodeS3Bucket,ParameterValue=gorillastack-autotag-releases-$REGION \\\n     ParameterKey=CodeS3Path,ParameterValue=autotag-0.5.1.zip \\\n     ParameterKey=AutoTagDebugLogging,ParameterValue=Disabled \\\n     ParameterKey=AutoTagTagsCreateTime,ParameterValue=Enabled \\\n     ParameterKey=AutoTagTagsInvokedBy,ParameterValue=Enabled \\\n  --capabilities CAPABILITY_NAMED_IAM \\\n  --region $REGION\n```\n\n### Deploy the IAM Role CloudFormation template\n\nIn each *additional* AWS account where tagging will be applied, deploy this IAM Role CloudFormation template in a single region. \n\n```bash\nexport REGION=ap-southeast-2               # set this to the region you plan to deploy to\nexport MAIN_AWS_ACCOUNT_NUMBER=11111111111 # set this to the AWS account number where we deployed the Main CloudFormation template\n\nwget https://raw.githubusercontent.com/GorillaStack/retro-tag/master/cloud_formation/autotag_retro_role-template.json\n\naws cloudformation create-stack \\\n  --template-body file://autotag_retro_role-template.json \\\n  --stack-name AutoTagRetro \\\n  --parameters \\\n     ParameterKey=MainAwsAccountNumber,ParameterValue=$MAIN_AWS_ACCOUNT_NUMBER \\\n     ParameterKey=MainStackName,ParameterValue=AutoTagRetro \\\n  --capabilities CAPABILITY_NAMED_IAM \\\n  --region $REGION\n```\n\n### Tag Existing Resources\n\nUse the `retro_tagging/retro_tag.rb` script to scan your environment for resources and then apply tagging to any resources that still exist.\n\nThe script will start by scanning each region for the AWS resources that exist then it will run the AutoTag lambda function against each CloudTrail log in S3 that includes at least one of the existing AWS resources.\n\n```bash\n$ bundle install # run this once to grab the ruby gem dependencies\nBundle complete! 17 Gemfile dependencies, 28 gems now installed.\n\nexport CSV_PATH=\"~/Downloads/MyAwsAccount_10292019.csv\" # set this to the CSV exported from Athena\nexport BUCKET=my-cloudtrail-bucket  # set this to the name of the CloudTrail S3 bucket\nexport BUCKET_REGION=ap-southeast-2 # set this to the region of the CloudTrail S3 bucket\nexport SCAN_PROFILE=development     # set this to a profile of the account where tagging will be applied, this should match the data in the CSV\nexport LAMBDA_PROFILE=development   # set this to a profile of the account where the Main CloudFormation template was deployed\nexport LAMBDA_REGION=ap-southeast-2 # set this to the region where the Main CloudFormation template was deployed\n\n./retro_tag.rb \\\n  --csv \"$CSV_PATH\" \\\n  --bucket $BUCKET \\\n  --bucket-region $BUCKET_REGION \\\n  --scan-profile \"$SCAN_PROFILE\" \\\n  --lambda-profile \"$LAMBDA_PROFILE\" \\\n  --lambda-region $LAMBDA_REGION\n```\n\n## Audit AutoTags\n\nUse the `retro_tagging/audit_tag.rb` script to scan all supported resources for Auto Tags to view the overall coverage of the Retro Tag process.\n\nThe script will start by scanning each region for the AWS resources that exist and show a report. \n\n`./audit_all_tags.rb --profile development-readonly`\n\n`./audit_all_tags.rb --access_key_id XXX --secret-access-key XXXXXX`\n\nEach resource's tags are inspected for the existence of the `AutoTag_Creator` and `AutoTag_CreateTime` required tags. For each AWS resource a point is added to either the `Passed` or `Failed` column based on each of those required tags existence.\n\nExample Output:\n\n```json\n+---------------------------+--------+--------+----------+\n|                 Auto-Tag Audit Summary                 |\n+---------------------------+--------+--------+----------+\n| Service                   | Passed | Failed | Coverage |\n+---------------------------+--------+--------+----------+\n| AutoScaling Groups        |     62 |      6 |   91.18% |\n+---------------------------+--------+--------+----------+\n| CloudWatch Alarms         |  2,034 |  1,904 |   51.65% |\n+---------------------------+--------+--------+----------+\n| CloudWatch Events Rules   |     64 |      0 |   100.0% |\n+---------------------------+--------+--------+----------+\n| CloudWatch Log Groups     |    680 |    110 |   86.08% |\n+---------------------------+--------+--------+----------+\n| Data Pipelines            |     82 |     14 |   85.42% |\n+---------------------------+--------+--------+----------+\n| DynamoDB Tables           |    564 |    148 |   79.21% |\n+---------------------------+--------+--------+----------+\n| EC2 AMIs                  |    178 |     30 |   85.58% |\n+---------------------------+--------+--------+----------+\n| EC2 Customer Gateways     |      2 |      6 |    25.0% |\n+---------------------------+--------+--------+----------+\n| EC2 DHCP Options Sets     |     10 |     46 |   17.86% |\n+---------------------------+--------+--------+----------+\n| EC2 EIPs                  |     52 |    124 |   29.55% |\n+---------------------------+--------+--------+----------+\n| EC2 Instances             |    282 |     48 |   85.45% |\n+---------------------------+--------+--------+----------+\n| EC2 Snapshots             |    260 |     90 |   74.29% |\n+---------------------------+--------+--------+----------+\n| EC2 Volumes               |    452 |     58 |   88.63% |\n+---------------------------+--------+--------+----------+\n| EMR Clusters              |      2 |      0 |   100.0% |\n+---------------------------+--------+--------+----------+\n| Elastic Load Balancing    |     98 |     38 |   72.06% |\n+---------------------------+--------+--------+----------+\n| Elastic Load Balancing V2 |      2 |      0 |   100.0% |\n+---------------------------+--------+--------+----------+\n| IAM Roles                 |    338 |     90 |   78.97% |\n+---------------------------+--------+--------+----------+\n| IAM Users                 |    276 |     46 |   85.71% |\n+---------------------------+--------+--------+----------+\n| Lambda Functions          |    152 |      0 |   100.0% |\n+---------------------------+--------+--------+----------+\n| OpsWorks Stacks           |     16 |      4 |    80.0% |\n+---------------------------+--------+--------+----------+\n| RDS Instances             |     25 |     12 |   67.57% |\n+---------------------------+--------+--------+----------+\n| S3 Buckets                |    158 |    170 |   48.17% |\n+---------------------------+--------+--------+----------+\n| Security Groups           |    994 |    514 |   65.92% |\n+---------------------------+--------+--------+----------+\n| VPC ENIs                  |    602 |    112 |   84.31% |\n+---------------------------+--------+--------+----------+\n| VPC Internet Gateways     |     60 |     20 |    75.0% |\n+---------------------------+--------+--------+----------+\n| VPC NAT Gateways          |     24 |      4 |   85.71% |\n+---------------------------+--------+--------+----------+\n| VPC Network ACLs          |     12 |     84 |    12.5% |\n+---------------------------+--------+--------+----------+\n| VPC Peering Connections   |     52 |      8 |   86.67% |\n+---------------------------+--------+--------+----------+\n| VPC Route Tables          |    166 |    116 |   58.87% |\n+---------------------------+--------+--------+----------+\n| VPC Subnets               |    380 |     84 |    81.9% |\n+---------------------------+--------+--------+----------+\n| VPCs                      |     62 |     18 |    77.5% |\n+---------------------------+--------+--------+----------+\n| VPN Connections           |      8 |      2 |    80.0% |\n+---------------------------+--------+--------+----------+\n| VPN Gateways              |     24 |     14 |   63.16% |\n+---------------------------+--------+--------+----------+\n```\n\n## FAQ\n### Supported Ruby versions\nThe retro tagging and audit logging functions were confirmed to be working on Ruby version 2.6.6.\n\n## Contributing\n\nIf you have questions, feature requests or bugs to report, please do so on [the issues section of our github repository](https://github.com/GorillaStack/retro-tag/issues).\n\nIf you are interested in contributing, please get started by forking our GitHub repository and submit a pull-request.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSWO-GS%2Fretro-tag","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FSWO-GS%2Fretro-tag","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSWO-GS%2Fretro-tag/lists"}