{"id":13401088,"url":"https://github.com/joshlarsen/aws-recon","last_synced_at":"2025-03-14T06:32:12.595Z","repository":{"id":37668631,"uuid":"277947677","full_name":"joshlarsen/aws-recon","owner":"joshlarsen","description":"Multi-threaded AWS inventory collection tool with a focus on security-relevant resources and metadata.","archived":false,"fork":false,"pushed_at":"2025-01-14T11:22:39.000Z","size":275,"stargazers_count":541,"open_issues_count":0,"forks_count":50,"subscribers_count":15,"default_branch":"main","last_synced_at":"2025-03-08T14:35:42.348Z","etag":null,"topics":["aws","cli","collection","inventory","scanner","security"],"latest_commit_sha":null,"homepage":"https://archive.darkbit.io/resources","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joshlarsen.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-08T00:03:02.000Z","updated_at":"2025-03-03T19:18:41.000Z","dependencies_parsed_at":"2023-07-14T10:49:45.259Z","dependency_job_id":"b9fe3c2b-d35f-4b0f-95de-d87cb781bf98","html_url":"https://github.com/joshlarsen/aws-recon","commit_stats":{"total_commits":315,"total_committers":6,"mean_commits":52.5,"dds":0.2063492063492064,"last_synced_commit":"d12c3b950c349c0ae571bcce97c2eed2372c9ea3"},"previous_names":["darkbitio/aws-recon"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joshlarsen%2Faws-recon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joshlarsen%2Faws-recon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joshlarsen%2Faws-recon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joshlarsen%2Faws-recon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joshlarsen","download_url":"https://codeload.github.com/joshlarsen/aws-recon/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243538060,"owners_count":20307100,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","cli","collection","inventory","scanner","security"],"created_at":"2024-07-30T19:00:58.558Z","updated_at":"2025-03-14T06:32:12.586Z","avatar_url":"https://github.com/joshlarsen.png","language":"Ruby","funding_links":["https://paypal.com/"],"categories":["Uncategorized","Ruby"],"sub_categories":["Uncategorized"],"readme":"[![Docker Pulls](https://img.shields.io/docker/pulls/darkbitio/aws_recon?logo=docker)](https://hub.docker.com/r/darkbitio/aws_recon)\n[![Gem Version](https://badge.fury.io/rb/aws_recon.svg)](https://rubygems.org/gems/aws_recon)\n[![GitHub Workflow Status (branch)](https://img.shields.io/github/workflow/status/joshlarsen/aws-recon/smoke-test/main)](https://github.com/joshlarsen/aws-recon/actions?query=branch%3Amain)\n[![AWS Service Regions](https://github.com/joshlarsen/aws-recon/actions/workflows/check-aws-regions.yml/badge.svg?branch=main\u0026event=schedule)](https://github.com/joshlarsen/aws-recon/actions/workflows/check-aws-regions.yml)\n\n# AWS Recon\n\nA multi-threaded AWS security-focused inventory collection tool written in Ruby.\n\nThis tool was created to facilitate efficient collection of a large amount of AWS resource attributes and metadata. It aims to collect nearly everything that is relevant to the security configuration and posture of an AWS environment.\n\nExisting tools (e.g. [AWS Config](https://aws.amazon.com/config)) that do some form of resource collection lack the coverage and specificity to accurately measure security posture (e.g. detailed resource attribute data, fully parsed policy documents, and nested resource relationships).\n\nAWS Recon handles collection from large accounts by taking advantage of automatic retries (either due to network reliability or API throttling), automatic paging of large responses (\u003e 100 resources per API call), and multi-threading parallel requests to speed up collection.\n\n## Project Goals\n\n- More complete resource coverage than available tools (especially for ECS \u0026 EKS)\n- More granular resource detail, including nested related resources in the output\n- Flexible output (console, JSON lines, plain JSON, file, S3 bucket, and standard out)\n- Efficient (multi-threaded, rate limited, automatic retries, and automatic result paging)\n- Easy to maintain and extend\n\n## Awesome companies using AWS Recon\\*\\*\n\n- [Netflix](https://www.netflix.com/)\n- [HashiCorp](https://www.hashicorp.com/)\n- [Workday](https://www.workday.com/)\n- [Stripe](https://stripe.com/)\n- [PayPal](https://paypal.com/)\n- [Typeform](https://typeform.com/)\n- [Amazon Web Services](https://aws.amazon.com/)\n- [Plaid](https://plaid.com/)\n- [Expel](https://expel.io/)\n- [Mozilla](https://www.mozilla.org/)\n- [Bugcrowd](https://www.bugcrowd.com/)\n- [Dropbox](https://www.dropbox.com/)\n- [Pinterest](https://www.pinterest.com/)\n- [HackerOne](https://www.hackerone.com/)\n- [MuleSoft](https://www.mulesoft.com/)\n- [Slack](https://slack.com/)\n- [Drata](https://drata.com/)\n- [Google](https://www.google.com/)\n- [Sophos](https://www.sophos.com/)\n- [Sumo Logic](https://www.sumologic.com/)\n- [Coalfile](https://www.coalfire.com/)\n- [Xero](https://www.xero.com/)\n\n\u003e \\*\\* usage does not imply endorsement\n\n## Setup\n\n### Requirements\n\nAWS Recon needs an AWS account role or credentials with `ReadOnlyAccess`. Full `AdministratorAccess` is over-privileged, but will work as well. The `SecurityAudit` policy is **not** sufficient as it omits access to many services.\n\n#### Running via Docker\n\nUse Docker version 19.x or above to run the pre-built image without having to install anything.\n\n#### Running locally via Ruby\n\nIf you already have Ruby installed (2.6.x or 2.7.x), you may want to install the Ruby gem.\n\n### Installation\n\nAWS Recon can be run locally via a Docker container or by installing the Ruby gem.\n\nTo run via a Docker a container, pass the necessary AWS credentials into the Docker `run` command. For example:\n\n```\n$ docker run -t --rm \\\n  -e AWS_REGION \\\n  -e AWS_ACCESS_KEY_ID \\\n  -e AWS_SECRET_ACCESS_KEY \\\n  -e AWS_SESSION_TOKEN \\\n  -v $(pwd)/output.json:/recon/output.json \\\n  darkbitio/aws_recon:latest \\\n  aws_recon -v -s EC2 -r global,us-east-1,us-east-2\n```\n\nTo run locally, first install the gem:\n\n```\n$ gem install aws_recon\nFetching aws_recon-0.5.17.gem\nFetching aws-sdk-3.0.1.gem\nFetching parallel-1.20.1.gem\n...\nSuccessfully installed aws-sdk-3.0.1\nSuccessfully installed parallel-1.20.1\nSuccessfully installed aws_recon-0.5.17\n```\n\nOr add it to your Gemfile using `bundle`:\n\n```\n$ bundle add aws_recon\nFetching gem metadata from https://rubygems.org/\nResolving dependencies...\n...\nUsing aws-sdk 3.0.1\nUsing parallel-1.20.1\nUsing aws_recon 0.5.17\n```\n\n## Usage\n\nAWS Recon will leverage any AWS credentials (see [requirements](#requirements)) currently available to the environment it runs in. If you are collecting from multiple accounts, you may want to leverage something like [aws-vault](https://github.com/99designs/aws-vault) to manage different credentials.\n\n```\n$ aws-vault exec profile -- aws_recon\n```\n\nPlain environment variables will work fine too.\n\n```\n$ AWS_PROFILE=\u003cprofile\u003e aws_recon\n```\n\nTo run from a Docker container using `aws-vault` managed credentials (output to stdout):\n\n```\n$ aws-vault exec \u003cvault_profile\u003e -- docker run -t --rm \\\n  -e AWS_REGION \\\n  -e AWS_ACCESS_KEY_ID \\\n  -e AWS_SECRET_ACCESS_KEY \\\n  -e AWS_SESSION_TOKEN \\\n  darkbitio/aws_recon:latest \\\n  aws_recon -j -s EC2 -r global,us-east-1,us-east-2\n```\n\nTo run from a Docker container using `aws-vault` managed credentials and output to a file, you will need to satisfy a couple of requirements. First, Docker needs access to bind mount the path you specify (or a parent path above). Second, you need to create an empty file to save the output into (e.g. `output.json`). This is because only that one file is mounted into the Docker container at run time. For example:\n\nCreate an empty file.\n\n```\n$ touch output.json\n```\n\nRun the `aws_recon` container, specifying the output file.\n\n```\n$ aws-vault exec \u003cvault_profile\u003e -- docker run -t --rm \\\n  -e AWS_REGION \\\n  -e AWS_ACCESS_KEY_ID \\\n  -e AWS_SECRET_ACCESS_KEY \\\n  -e AWS_SESSION_TOKEN \\\n  -v $(pwd)/output.json:/recon/output.json \\\n  darkbitio/aws_recon:latest \\\n  aws_recon -s EC2 -v -r global,us-east-1,us-east-2\n```\n\nYou may want to use the `-v` or `--verbose` flag initially to see status and activity while collection is running.\n\nIn verbose mode, the console output will show:\n\n```\n\u003cthread\u003e.\u003cregion\u003e.\u003cservice\u003e.\u003coperation\u003e\n```\n\nThe `t` prefix indicates which thread a particular request is running under. Region, service, and operation indicate which request operation is currently in progress and where.\n\n```\n$ aws_recon -v\n\nt0.global.EC2.describe_account_attributes\nt2.global.S3.list_buckets\nt3.global.Support.describe_trusted_advisor_checks\nt2.global.S3.list_buckets.acl\nt5.ap-southeast-1.WorkSpaces.describe_workspaces\nt6.ap-northeast-1.Lightsail.get_instances\n...\nt2.us-west-2.WorkSpaces.describe_workspaces\nt1.us-east-2.Lightsail.get_instances\nt4.ap-southeast-1.Firehose.list_delivery_streams\nt7.ap-southeast-1.Lightsail.get_instances\nt0.ap-south-1.Lightsail.get_instances\nt1.us-east-2.Lightsail.get_load_balancers\nt7.ap-southeast-2.WorkSpaces.describe_workspaces\nt2.eu-west-3.SageMaker.list_notebook_instances\nt3.eu-west-2.SageMaker.list_notebook_instances\n\nFinished in 46 seconds. Saving resources to output.json.\n```\n\n#### Example command line options\n\n```\n# collect S3 and EC2 global resources, as well as us-east-1 and us-east-2\n\n$ AWS_PROFILE=\u003cprofile\u003e aws_recon -s S3,EC2 -r global,us-east-1,us-east-2\n```\n\n```\n# collect S3 and EC2 global resources, as well as us-east-1 and us-east-2\n\n$ AWS_PROFILE=\u003cprofile\u003e aws_recon --services S3,EC2 --regions global,us-east-1,us-east-2\n```\n\n```\n# save output to S3 bucket\n\n$ AWS_PROFILE=\u003cprofile\u003e aws_recon \\\n  --services S3,EC2 \\\n  --regions global,us-east-1,us-east-2 \\\n  --verbose \\\n  --s3-bucket my-recon-bucket\n```\n\n```\n# save output to S3 bucket with a home region other than us-east-1\n\n$ AWS_PROFILE=\u003cprofile\u003e aws_recon \\\n  --services S3,EC2 \\\n  --regions global,us-east-1,us-east-2 \\\n  --verbose \\\n  --s3-bucket my-recon-bucket:us-west-2\n```\n\nExample [OpenCSPM](https://github.com/OpenCSPM/opencspm) formatted (NDJSON) output.\n\n```\n$ AWS_PROFILE=\u003cprofile\u003e aws_recon -l \\\n  -s S3,EC2 \\\n  -r global,us-east-1,us-east-2 \\\n  -f custom\n```\n\nor\n\n```\n$ AWS_PROFILE=\u003cprofile\u003e aws_recon -j \\\n  -s S3,EC2 \\\n  -r global,us-east-1,us-east-2 \\\n  -f custom \u003e output.json\n```\n\n#### Errors\n\nAPI exceptions related to permissions are silently ignored in most cases. These errors are usually due to one of these cases:\n\n- using a role without sufficient permissions\n- querying an account with SCPs in place that prevent usage of certain services\n- trying to query a service that isn't enabled/available in your region/account\n\nIn `verbose` mode, you will see exception logs in the output:\n\n```\nt2.us-east-1.EC2.describe_subnets.0\nt4.us-east-1.SSM.describe_instance_information.0\nt6.us-east-1.SecurityHub.InvalidAccessException   \u003c-----\nt2.us-east-1.EC2.describe_addresses.0\nt4.us-east-1.SSM.describe_parameters.0\nt1.us-east-1.GuardDuty.list_detectors.0\n```\n\nUse the `-q` command line option to re-raise these exceptions so troubleshooting access issues is easier.\n\n```\nTraceback (most recent call last):\narn:aws:sts::1234567890:assumed-role/role/my-audit-role is not authorized to perform:\n codepipeline:GetPipeline on resource: arn:aws:codepipeline:us-west-2:1234567890:pipeline\n (Aws::CodePipeline::Errors::AccessDeniedException)\n```\n\nThe exact API operation that triggered the exception is indicated on the last line of the stack trace. If you can't resolve the necessary access, you should exclude those services with `-x` or `--not-services`, or leave off the `-q` option so the collection can continue.\n\n### Threads\n\nAWS Recon uses multiple threads to try to overcome some of the I/O challenges of performing many API calls to endpoints all over the world.\n\nFor global services like IAM, Shield, and Support, requests are not multi-threaded. The S3 module is multi-threaded since each bucket requires several additional calls to collect complete metadata.\n\nFor regional services, a thread (up to the thread limit) is spawned for each service in a region. By default, up to 8 threads will be used. If your account has resources spread across many regions, you may see a speed improvement by increasing threads with `-t X`, where `X` is the number of threads.\n\n### Performance\n\nAWS Recon will make a minimum of ~2,000 API calls in a new/empty account, just to query the supported services in all 20 standard (non-GovCloud, non-China) regions. It is very likely to encounter API rate-limiting (throttling) on large accounts if you enable more threads than the default (8).\n\nRecon will automatically backoff and respect the retry limits in the API response. If you observe long pauses during collection, this is likely what is happening. Retry collection with the `-d` or `--debug` option to observe the wire trace and see if you're being throttled. Consider using fewer threads or requesting higher rate limits from AWS if you are regularly getting rate-limited.\n\n### Options\n\nMost users will want to limit collection to relevant services and regions. Running without any exclusions will attempt to collect all resources from all regions enabled for the account.\n\n```\n$ aws_recon -h\n\nAWS Recon - AWS Inventory Collector (0.5.17)\n\nUsage: aws_recon [options]\n    -r, --regions [REGIONS]          Regions to scan, separated by comma (default: all)\n    -n, --not-regions [REGIONS]      Regions to skip, separated by comma (default: none)\n    -s, --services [SERVICES]        Services to scan, separated by comma (default: all)\n    -x, --not-services [SERVICES]    Services to skip, separated by comma (default: none)\n    -c, --config [CONFIG]            Specify config file for services \u0026 regions (e.g. config.yaml)\n    -b, --s3-bucket [BUCKET:REGION]  Write output file to S3 bucket (default: '')\n    -o, --output [OUTPUT]            Specify output file (default: output.json)\n    -f, --format [FORMAT]            Specify output format (default: aws)\n    -t, --threads [THREADS]          Specify max threads (default: 8, max: 128)\n    -l, --json-lines                 Output NDJSON/JSONL format (default: false)\n    -u, --user-data                  Collect EC2 instance user data (default: false)\n    -z, --skip-slow                  Skip slow operations (default: false)\n    -g, --skip-credential-report     Skip generating IAM credential report (default: false)\n    -j, --stream-output              Stream JSON lines to stdout (default: false)\n    -v, --verbose                    Output client progress and current operation\n    -q, --quit-on-exception          Stop collection if an API error is encountered (default: false)\n    -d, --debug                      Output debug with wire trace info\n    -h, --help                       Print this help information\n\n```\n\n#### Output\n\nOutput is always some form of JSON - either JSON lines or plain JSON. The output is either written to a file (the default), or written to stdout (with `-j`).\n\nWhen writing to an S3 bucket, the JSON output is automatically compressed with `gzip`.\n\n## Support for Manually Enabled Regions\n\nIf you have enabled **manually enabled regions**:\n\n- me-south-1 - Middle East (Bahrain)\n- af-south-1 - Africa (Cape Town)\n- ap-east-1 - Asia Pacific (Hong Kong)\n- eu-south-1 - Europe (Milan)\n\nand you are using STS to assume a role into an account, you will need to [enable v2 STS tokens](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html) in the account you are assuming the role **from** to be able to run AWS Recon against those regions.\n\n\u003e Version 1 tokens are valid only in AWS Regions that are available by default. These tokens do not work in manually enabled Regions, such as Asia Pacific (Hong Kong). Version 2 tokens are valid in all Regions. However, version 2 tokens are longer and might affect systems where you temporarily store tokens.\n\nIf you are using a static access key/secret, you can collect from these regions regardless of STS token version.\n\n## Supported Services \u0026 Resources\n\nCurrent \"coverage\" by service is listed below. The services without coverage will eventually be added. PRs are certainly welcome. :)\n\nAWS Recon aims to collect all resources and metadata that are relevant in determining the security posture of your AWS account(s). However, it does not actually examine the resources for security posture - that is the job of other tools that take the output of AWS Recon as input.\n\n- [x] AccessAnalyzer\n- [x] AdvancedShield\n- [x] ApplicationAutoScaling\n- [x] Athena\n- [x] Backup\n- [x] GuardDuty\n- [ ] Macie\n- [x] Systems Manager\n- [x] Trusted Advisor\n- [x] ACM\n- [x] API Gateway\n- [x] AutoScaling\n- [x] CodePipeline\n- [x] CodeBuild\n- [x] CloudFormation\n- [x] CloudFront\n- [x] CloudWatch\n- [x] CloudWatch Logs\n- [x] CloudTrail\n- [x] Config\n- [x] DirectoryService\n- [x] DirectConnect\n- [x] DMS\n- [x] DynamoDB\n- [x] EC2\n- [x] ECR\n- [x] ECRPublic\n- [x] ECS\n- [x] EFS\n- [x] EKS\n- [x] ELB\n- [x] EMR\n- [x] Elasticsearch\n- [x] ElastiCache\n- [x] Firehose\n- [ ] FMS\n- [ ] Glacier\n- [x] Glue\n- [x] IAM\n- [x] KMS\n- [x] Kafka\n- [x] Kinesis\n- [x] Lambda\n- [x] Lightsail\n- [x] Organizations\n- [x] RDS\n- [x] Redshift\n- [x] Route53\n- [x] Route53Domains\n- [x] S3\n- [x] SageMaker\n- [x] SES\n- [x] SecretsManager\n- [x] SecurityHub\n- [x] ServiceQuotas\n- [x] Shield\n- [x] SNS\n- [x] SQS\n- [x] Transfer\n- [x] VPC\n- [ ] WAF\n- [x] WAFv2\n- [x] Workspaces\n- [x] Xray\n\n### Additional Coverage\n\nOne of the primary motivations for AWS Recon was to build a tool that is easy to maintain and extend. If you feel like coverage could be improved for a particular service, we would welcome PRs to that effect. Anyone with a moderate familiarity with Ruby will be able to mimic the pattern used by the existing collectors to query a specific service and add the results to the resource collection.\n\n### Development\n\nClone this repository:\n\n```\n$ git clone git@github.com:darkbitio/aws-recon.git\n$ cd aws-recon\n```\n\nCreate a sticky gemset if using RVM:\n\n```\n$ rvm use 2.7.2@aws_recon_dev --create --ruby-version\n```\n\nRun `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.\n\nTo install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).\n\n### TODO\n\n- [ ] Test coverage with AWS SDK stubbed resources\n\n## Kudos\n\nAWS Recon was inspired by the excellent work of the people and teams behind these tools:\n\n- CloudMapper [https://github.com/duo-labs/cloudmapper](https://github.com/duo-labs/cloudmapper)\n- Prowler [https://github.com/toniblyx/prowler](https://github.com/toniblyx/prowler)\n- CloudSploit [https://github.com/cloudsploit/scans](https://github.com/cloudsploit/scans)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoshlarsen%2Faws-recon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoshlarsen%2Faws-recon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoshlarsen%2Faws-recon/lists"}