{"id":13510684,"url":"https://github.com/Cloud-Architects/cloudiscovery","last_synced_at":"2025-03-30T16:34:43.030Z","repository":{"id":37253290,"uuid":"260712454","full_name":"Cloud-Architects/cloudiscovery","owner":"Cloud-Architects","description":"The tool to help you discover resources in the cloud environment","archived":false,"fork":false,"pushed_at":"2024-06-18T03:08:56.000Z","size":3561,"stargazers_count":770,"open_issues_count":31,"forks_count":92,"subscribers_count":22,"default_branch":"develop","last_synced_at":"2024-10-13T14:56:54.112Z","etag":null,"topics":["aws","aws-cli","aws-iot","aws-monitoring","aws-services","devops-tools","diagrams","iam-policy","iot","python","vpc"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Cloud-Architects.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-05-02T15:07:50.000Z","updated_at":"2024-10-09T17:14:57.000Z","dependencies_parsed_at":"2024-01-13T19:35:27.678Z","dependency_job_id":"2138463d-0feb-4454-983a-f24ff8144845","html_url":"https://github.com/Cloud-Architects/cloudiscovery","commit_stats":{"total_commits":378,"total_committers":3,"mean_commits":126.0,"dds":"0.43386243386243384","last_synced_commit":"fad132e45f813775eaf0051e628fcbec68291fb4"},"previous_names":["cloud-architects/cloud-discovery"],"tags_count":23,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Cloud-Architects%2Fcloudiscovery","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Cloud-Architects%2Fcloudiscovery/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Cloud-Architects%2Fcloudiscovery/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Cloud-Architects%2Fcloudiscovery/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Cloud-Architects","download_url":"https://codeload.github.com/Cloud-Architects/cloudiscovery/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222566739,"owners_count":17004237,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-cli","aws-iot","aws-monitoring","aws-services","devops-tools","diagrams","iam-policy","iot","python","vpc"],"created_at":"2024-08-01T02:01:49.695Z","updated_at":"2024-11-01T11:31:06.851Z","avatar_url":"https://github.com/Cloud-Architects.png","language":"Python","readme":"# Cloudiscovery\n\n[![PyPI version](https://badge.fury.io/py/cloudiscovery.svg)](https://badge.fury.io/py/cloudiscovery)\n[![Downloads](https://pepy.tech/badge/cloudiscovery)](https://pepy.tech/project/cloudiscovery)\n[![codecov](https://codecov.io/gh/Cloud-Architects/cloudiscovery/branch/develop/graph/badge.svg)](https://codecov.io/gh/Cloud-Architects/cloudiscovery)\n![python version](https://img.shields.io/badge/python-3.8-blue?logo=python)\n[![CircleCI](https://circleci.com/gh/Cloud-Architects/cloudiscovery.svg?style=svg)](https://circleci.com/gh/Cloud-Architects/cloudiscovery)\n[![Codacy Badge](https://app.codacy.com/project/badge/Grade/c0a7a5bc51044c7ca8bd9115965e4467)](https://www.codacy.com/gh/Cloud-Architects/cloudiscovery?utm_source=github.com\u0026amp;utm_medium=referral\u0026amp;utm_content=Cloud-Architects/cloudiscovery\u0026amp;utm_campaign=Badge_Grade)\n[![Hits-of-Code](https://sloc.xyz/github/Cloud-Architects/cloudiscovery?category=code)](https://github.com/Cloud-Architects/cloudiscovery)\n[![GitHub license](https://img.shields.io/github/license/Cloud-Architects/cloudiscovery.svg)](https://github.com/Cloud-Architects/cloudiscovery/blob/develop/LICENSE)\n\n![aws provider](https://img.shields.io/badge/provider-AWS-orange?logo=amazon-aws\u0026color=ff9900)\n\nCloudiscovery helps you to analyze resources in your cloud (AWS/GCP/Azure/Alibaba/IBM) account. Now this tool only can check resources in AWS, but we are working to expand to other providers.\n\nThe tool consists of various commands to help you understand the cloud infrastructure.\n\n## Features\n\n### Diagrams\n\nCommands can generate diagrams. When modelling them, we try to follow the principle:\n\n\u003e Graphical excellence is that which gives to the viewer the greatest number of ideas in the shortest time with the least ink in the smallest space.\n\nEdward Tufte\n\n### Report\n\nThe commands generate reports that can be used to further analyze resources.\n\n### CLI\n\n1.  Run the cloudiscovery command with following options (if a region not pass, this script will try to get it from ~/.aws/credentials):\n\n1.1 To detect AWS VPC resources (more on [AWS VPC](#aws-vpc)):\n\n```sh\ncloudiscovery aws-vpc [--vpc-id vpc-xxxxxxx] --region-name xx-xxxx-xxx [--profile-name profile] [--diagram [yes/no]] [--filter xxx] [--verbose]\n```\n1.2 To detect AWS policy resources (more on [AWS Policy](#aws-policy)):\n\n```sh\ncloudiscovery aws-policy [--profile-name profile] [--diagram [yes/no]] [--filter xxx] [--verbose]\n```\n1.3 To detect AWS IoT resources (more on [AWS IoT](#aws-iot)):\n\n```sh\ncloudiscovery aws-iot [--thing-name thing-xxxx] --region-name xx-xxxx-xxx [--profile-name profile] [--diagram [yes/no]] [--filter xxx] [--verbose]\n```\n\n1.4 To detect all AWS resources (more on [AWS All](#aws-all)):\n\n```sh\ncloudiscovery aws-all --region-name xx-xxxx-xxx [--profile-name profile] [--services xxx,xxx] [--filter xxx] [--verbose]\n```\n\n1.5 To check AWS limits per resource (more on [AWS Limit](#aws-limit)):\n\n```sh\ncloudiscovery aws-limit --region-name xx-xxxx-xxx [--profile-name profile] [--services xxx,xxx] [--usage 0-100] [--verbose]\n```\n\n1.6 To run AWS security controls (experimental feature):\n\n```sh\ncloudiscovery aws-security --region-name xx-xxxx-xxx [--profile-name profile] [--commands x] [--verbose]\n```\n\n2.  For help use:\n\n```sh\ncloudiscovery [aws-vpc|aws-policy|aws-iot|aws-all|aws-limit] -h\n```\n\n### Debbuging \n\nEnabling verbose mode, it is possible to debug all calls to the providers endpoints and check possible problems.\n\n### Filtering\n\nIt's possible to filter resources by tags and resource type. To filter, add an option `--filter \u003cVALUE\u003e`, where `\u003cVALUE\u003e` can be:\n\n1.  `Name=tags.costCenter;Value=20000` - to filter resources by a tag name `costCenter` and with value `20000`.\n2.  `Name=type;Value=aws_lambda_function` to only list lambda functions.\n\nIt's possible to pass multiple values, to be able to select a value from a set. Values are split by `:` sign. If a desired value has a `:` sign, wrap it in `'` signs e.g. `--filter=\"Name=tags.costCenter;Value=20000:'20001:1'`.\n\nIt is possible to pass multiple filter options, just pass `-f filter_1 -f filter_2`. In that case, the tool will return resources that match either of the filters\n\nUseful [CF tags](https://aws.amazon.com/blogs/devops/tracking-the-cost-of-your-aws-cloudformation-stack/):\n1.  `aws:cloudformation:stack-name` - Stack name\n2.  `aws:cloudformation:stack-id` - Stack id\n3.  `aws:cloudformation:logical-id` - Logical id defined in CF template\n\n## Requirements and Installation\n\n### Installation\n\nThis tool has been written in Python3+ and AWS-CLI and it works on Linux, Windows and Mac OS.\n\nMake sure the latest version of AWS-CLI is installed on your workstation, and other components needed, with Python pip already installed:\n\n```sh\npip install -U cloudiscovery\n```\n\nOnce a while after installation, there can be some issues related with a cache from older version being used by a newer version. In that case, it's recommended to remove directory `./assets/.cache`.\n\n### AWS Credentials\n\nMake sure you have properly configured your AWS-CLI with a valid Access Key and Region:\n\n```sh\naws configure\n```\n\nMore on credentials configuration: [Configuration basics](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html)\n\n#### AWS Permissions\n\nThe configured credentials must be associated to a user or role with proper permissions to do all checks. If you want to use a role with narrowed set of permissions just to perform cloud discovery, use a role from the following the [CF template maintained by our team](docs/assets/role-template.json). \n\nTo further increase security, you can add a block to check `aws:MultiFactorAuthPresent` condition in `AssumeRolePolicyDocument`. More on using IAM roles in the [configuration file](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-role.html).\n\n\n(Optional) If you want to be able to switch between multiple AWS credentials and settings, you can configure [named profiles](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html) and later pass profile name when running the tool.\n\n## Commands\nCloudiscovery provides a CLI to easily perform desired actions.\n\n### AWS VPC\nExample of a diagram ([diagrams.net](https://www.diagrams.net/) supported):\n\n![diagrams logo](docs/assets/aws-vpc.png)\n\nFollowing resources are checked in VPC command:\n\n*   Autoscaling Group\n*   Classic/Network/Application Load Balancer\n*   Client VPN Endpoints\n*   CloudHSM\n*   DocumentDB\n*   Directory Service\n*   EC2 Instance\n*   ECS\n*   EFS\n*   ElastiCache\n*   Elasticsearch\n*   EKS\n*   EMR \n*   IAM Policy\n*   Internet Gateway (IGW)\n*   Lambda\n*   Media Connect\n*   Media Live\n*   Media Store Policy\n*   MSK\n*   NACL\n*   NAT Gateway\n*   Neptune\n*   QuickSight\n*   RDS\n*   REST Api Policy\n*   Route Table\n*   S3 Policy\n*   Sagemaker Notebook\n*   Sagemaker Training Job\n*   Sagemaker Model\n*   Security Group\n*   SQS Queue Policy\n*   Site-to-Site VPN Connections\n*   Subnet\n*   Synthetic Canary\n*   VPC Peering\n*   VPC Endpoint\n*   VPN Customer Gateways\n*   Virtual Private Gateways\n*   Workspace\n\nThe subnets are aggregated to simplify the diagram and hide infrastructure redundancies. There can be two types of subnet aggregates:\n1.  Private*   ones with a route `0.0.0.0/0` to Internet Gateway\n2.  Public*   ones without any route to IGW\n\nIf EC2 instances and ECS instances are part of an autoscaling group, those instances will be aggregated on a diagram.\n\nMore information: [AWS WA, REL 2: How do you plan your network topology?](https://wa.aws.amazon.com/wat.question.REL_2.en.html)\n\n### AWS Policy\nExample of a diagram:\n\n![diagrams logo](docs/assets/aws-policy.png)\n\nFollowing resources are checked in Policy command:\n\n*   [AWS Principal](https://gist.github.com/shortjared/4c1e3fe52bdfa47522cfe5b41e5d6f22) that are able to assume roles\n*   IAM Group\n*   IAM Group to policy relationship\n*   IAM Policy\n*   IAM Role\n*   IAM Role to policy relationship\n*   IAM User\n*   IAM User to group relationship\n*   IAM User to policy relationship\n\nSome roles can be aggregated to simplify the diagram. If a role is associated with a principal and is not attached to any named policy, will be aggregated.\n\nMore information: [AWS WA, SEC 3: How do you manage permissions for people and machines?](https://wa.aws.amazon.com/wat.question.SEC_3.en.html)\n\n### AWS IoT\nExample of a diagram:\n\n![diagrams logo](docs/assets/aws-iot.png)\n\nFollowing resources are checked in IoT command:\n\n*   IoT Billing Group\n*   IoT Certificates\n*   IoT Jobs\n*   IoT Policies\n*   IoT Thing\n*   IoT Thing Type\n\n### AWS All\nA command to list **ALL** AWS resources.\n\nExample of an HTML report:\n\n![diagrams logo](docs/assets/aws-all.png)\n\nThe command calls all AWS services (200+) and operations with name `Describe`, `Get...` and `List...` (500+).\n\nThe operations must be allowed to be called by permissions described in [AWS Permissions](#aws-permissions).\n\nTypes of resources mostly cover Terraform types. It is possible to narrow down scope of the resources to ones related with a given service with parameter `-s` e.g. `-s ec2,ecs,cloudfront,rds`.\n\nMore information: [AWS WA, COST 2: How do you govern usage?](https://wa.aws.amazon.com/wat.question.COST_2.en.html)\n\n### AWS Limit\nIt's possible to check resources limits  across various service in an account. This command implements over 60 limits checks.\n\nExample of an HTML report:\n\n![diagrams logo](docs/assets/aws-limit.png)\n\nWith `--services value,value,value` parameter, you can narrow down checks to just services that you want to check. \n\nWith `--threshold 0-100` option, you can customize a minimum percentage threshold to start reporting a warning.\n\n*   Services available\n    *   Acm\n    *   Amplify\n    *   Apigateway\n    *   Appmesh\n    *   Appsync\n    *   Autoscaling Plans\n    *   Batch\n    *   Chime\n    *   Code Artifact\n    *   Code Build\n    *   Code Commit\n    *   Code Deploy\n    *   Codeguru Reviewer\n    *   Codeguru Profiler\n    *   Cognito Federated Identities\n    *   Cloudformation\n    *   Cloud Map\n    *   CloudWatch Logs\n    *   Dynamodb\n    *   EBS\n    *   EC2\n    *   ECR\n    *   ECS\n    *   Elastic Inference\n    *   Elastic Filesystem\n    *   Elastic Beanstalk\n    *   Elastic Loadbalancing\n    *   Forecast\n    *   Fraud Detector\n    *   Gamelift\n    *   Glue\n    *   IAM\n    *   Inspector\n    *   Kendra\n    *   KMS\n    *   Media Connect\n    *   Media Live\n    *   Media Package\n    *   Metwork Manager\n    *   Polly\n    *   Qldb\n    *   Robomaker\n    *   Route53\n    *   Route53resolver\n    *   RDS\n    *   S3\n    *   SES\n    *   SNS\n    *   SWF\n    *   Transcribe\n    *   Translate\n    *   VPC\n\nAWS has a default quota to all services. At the first time that an account is created, AWS apply this default quota to all services.  \nAn administrator can ask to increase the quota value of a certain service via ticket. This command helps administrators detect those issues in advance.\n\nMore information: [AWS WA, REL 1 How do you manage service limits?](https://wa.aws.amazon.com/wat.question.REL_1.en.html)\n\n### AWS Security\nThis features is experimental, but now you can run commands to check and analyze some security issues. The following commands are available now:\n\n*   Access key age\n*   EBS Encryption enabled\n*   EC2 IMDSV2 Check\n*   DynamoDB PITR Enabled\n*   Incoming SSH Disabled\n*   Cloudtrail enabled\n\n\n## Regions outside of main partition\n\nIf you wish to analyze accounts in regions outside the main AWS partition (e.g. GovCloud or China), you should provide credentials (e.g. a profile) that are applicable to a given partition. It's not possible to analyze regions from multiple partitions.\n\n## Using a Docker container\nTo build docker container using Dockerfile\n\n```sh\ndocker build -t cloudiscovery .\n```\n\nAfter build container, you must start container using follow command. The run command will mount a filesystem with your actual aws cli credentials, then you won't need configure aws cli again.\n\n```sh\ndocker run \\\n-it \\\n--mount type=bind,source=$HOME/.aws/,target=/root/.aws/,readonly \\\ncloudiscovery \\\n/bin/bash\n\n```\n\n*   If you are using Diagram output and due to fact container is a slim image of Python image, you must run cloudiscovery with \"--diagram no\", otherwise you'll have an error about \"xdg-open\". The output file will be saved in \"assets/diagrams\".\n\n## Translate\nThis project support English and Portuguese (Brazil) languages. To contribute with a translation, follow this steps:\n\n*   Create a folder inside locales folder with prefix of new idiom with appropiate [locale code](https://docs.oracle.com/cd/E23824_01/html/E26033/glset.html). Copy \"locales/messages.pot\" to locales/newfolder/LC_MESSAGES/.\n*   To build \".mo\" file running this command from project root folder:\n\n```sh\npython msgfmt.py -o locales/NEWFOLDER/LC_MESSAGES/messages.mo locales/NEWFOLDER/LC_MESSAGES/messages\n```\n\n## Contributing\n\nIf you have improvements or fixes, we would love to have your contributions. Please use [PEP 8](https://pycodestyle.readthedocs.io/en/latest/) code style.\n\n## Development\n\nWhen developing, it's recommended to use [venv](https://docs.python.org/3/library/venv.html).\n\nIn order to create a venv on macOS and Linux:\n```sh\npython3 -m venv env\n```\nOn Windows:\n```sh\npy -m venv venv\nOR\npython -v venv venv\n```\nOnce installed, you need to activate the virtual environment. Activation will put specific paths for `python` and `pip` commands.\nOn macOS and Linux call:\n```sh\nsource venv/bin/activate\n```\nOn Windows:\n```sh\n.\\venv\\Scripts\\activate\n```\n\nMake sure you have installed [pre-commit](https://pre-commit.com/#installation).\n\nInstall development requirements:\n```sh\npip install -U -r requirements.txt -r requirements-dev.txt\n```\n\nAdd precommit hooks:\n```sh\npre-commit install\n```\n\nTo run pre-commit hooks, you can issue the following command:\n```sh\npre-commit run --all-files\n```\n\nRunning cloudiscovery in development mode:\n```\npython cloudiscovery/__init__.py OPTIONS\n```\n\nTo add new resources to check limit, please remove \"assets/.cache/cache.db\"\n\n## Making a release\n\n1.  Update the version in cloudiscovery/__init\\__.py and create a new git tag with `git tag $VERSION`.\n2.  Once you push the tag to GitHub with `git push --tags`, a new CircleCI build is triggered.\n\n## Similar projects and products\n\n*   [mingrammer/diagrams](https://github.com/mingrammer/diagrams) - library being used to draw diagrams\n*   [Lucidchart Cloud Insights](https://www.lucidchart.com/pages/solutions/cloud-insights) - commercial extension to Lucidchart\n*   [Cloudcraft](https://cloudcraft.co) - commercial visualization tool","funding_links":[],"categories":["Python","Other Awesome Lists","python","aws"],"sub_categories":["Visual Resource Graphing"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCloud-Architects%2Fcloudiscovery","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FCloud-Architects%2Fcloudiscovery","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCloud-Architects%2Fcloudiscovery/lists"}