https://github.com/yugabyte/cloud-resource-cleanup
Tool that allows you to delete and stop resources across different clouds
https://github.com/yugabyte/cloud-resource-cleanup
cleanup-tool cli-app cloud-cost cloud-cost-efficiency cloud-cost-intelligence cloud-cost-optimization cloud-costs cloud-resource-manager cloud-resources hybrid-cloud infrastructure leak-detection
Last synced: 3 months ago
JSON representation
Tool that allows you to delete and stop resources across different clouds
- Host: GitHub
- URL: https://github.com/yugabyte/cloud-resource-cleanup
- Owner: yugabyte
- License: apache-2.0
- Created: 2023-01-14T15:41:13.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2025-12-15T10:52:02.000Z (6 months ago)
- Last Synced: 2025-12-18T11:22:47.831Z (5 months ago)
- Topics: cleanup-tool, cli-app, cloud-cost, cloud-cost-efficiency, cloud-cost-intelligence, cloud-cost-optimization, cloud-costs, cloud-resource-manager, cloud-resources, hybrid-cloud, infrastructure, leak-detection
- Language: Python
- Homepage:
- Size: 372 KB
- Stars: 4
- Watchers: 69
- Forks: 1
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
Awesome Lists containing this project
README
---------------------------------------
[](https://opensource.org/licenses/Apache-2.0)
[](https://docs.yugabyte.com/)
[](https://forum.yugabyte.com/)
[](https://communityinviter.com/apps/yugabyte-db/register)
[](https://github.com/yugabyte/ga-beacon)
# cloud-resource-cleanup
Introducing `cloud-resource-cleanup` (`crc` for short), a powerful tool that allows you to easily **delete** and **stop** resources across different clouds.
* [Core Features](#core-features)
* [Get Started](#get-started)
* [Usage](#usage)
* [Examples](#examples)
* [Notes](#notes)
* [Need Help?](#need-help)
* [Contribute](#contribute)
* [License](#license)
* [Read More](#read-more)
# Core Features
We support below Cloud Providers:
* AWS
* Delete Orphan Elastic IPs
* Delete Orphan keypairs
* Delete VMs (including attached resources such as Disks and NICs)
* Stop VMs
* Delete KMS
* Delete Spot Instance Requests & associated Instances
* Azure
* Delete Orphan disks
* Delete VMs (and attached NICs)
* Delete Orphan public IPs
* Stop VMs
* GCP
* Delete Orphan Disks
* Delete Orphan IPs
* Delete VMs (including attached resources such as Disks and NICs)
* Stop VMs
In addition to these features, `cloud-resource-cleanup` also includes the following features:
* `Dry Run mode`: Preview resources that match your criteria without actually performing any operations on them
* `Slack notifications`: Get notifications on your Slack channel.
* `InfluxDB support`: Receive data on the execution of the script by specifying a connection to InfluxDB
* `Age-based filtering`: Filter resources based on their `age`, considering the timezone.
* `Tag-based filtering`: Choose to include resources with specific tags. (`--filter_tags` option)
* `Delete untagged resources`: Option to delete resources that don't have specified tags. (`--notags` option)
* `Keep tagged resources`: Option to keep resources that have specific tags. (`--exception_tags` option)
* `Custom Age Override`: Specify a custom age for a resource using the `--custom_age_tag_key` option.
Get started with `cloud-resource-cleanup` today and see the difference it can make for your cloud infrastructure management.
# Get Started
## Prerequisites
* Python 3.x
* Required python packages
* boto3 (for AWS)
* msrestazure (for Azure)
* azure-mgmt-compute (for Azure)
* azure-identity (for Azure)
* azure-mgmt-network (for Azure)
* google-cloud-compute (for GCP)
* google-api-python-client (for GCP)
## Installation
```
git clone https://github.com/yugabyte/cloud-resource-cleanup.git
cd cloud-resource-cleanup
pip install -r requirements.txt
```
## Environment Variables
The script requires certain environment variables to be set in order to interact with different cloud providers. The following environment variables must be set before running the script:
### Google Cloud Platform
* `GOOGLE_APPLICATION_CREDENTIALS`: The path to the JSON file containing your GCP service account credentials.
### Amazon Web Services
* `AWS_SECRET_ACCESS_KEY`: The secret access key for your AWS account.
* `AWS_ACCESS_KEY_ID`: The access key ID for your AWS account.
### Azure
* `AZURE_CREDENTIALS_TENANT_ID`: The tenant ID for your Azure subscription.
* `AZURE_CREDENTIALS_SUBSCRIPTION_ID`: The subscription ID for your Azure subscription.
* `AZURE_CREDENTIALS_CLIENT_SECRET`: The client secret for your Azure application.
* `AZURE_CREDENTIALS_CLIENT_ID`: The client ID for your Azure application.
* `AZURE_RESOURCE_GROUP`: The name of the resource group to use in Azure. If you don't want to set this as an environment variable or want to use a different resource group than the one set in the environment variable, you can specify it using the `--resource_group` option. Note that the `--resource_group` option will take priority over the environment variable.
### Slack (Optional)
* `SLACK_BOT_TOKEN`: The API token for the Slack bot you want to use to receive notifications.
### InfluxDB (Optional)
* `INFLUXDB_TOKEN`: The API token for the InfluxDB you want to use to receive data.
It's important to note that you only need to set the environment variables for the cloud providers you are interacting with. For example, if you are only using the script to delete resources on AWS, you would only need to set the `AWS_SECRET_ACCESS_KEY` and `AWS_ACCESS_KEY_ID` environment variables.
You can set the environment variables in your shell by using the `export` command. For example, to set the `GOOGLE_APPLICATION_CREDENTIALS` environment variable, you would use the following command:
```
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/credentials.json"
```
You can also add these commands to your shell profile file, such as `~/.bash_profile` or `~/.bashrc`, to ensure that these environment variables are set every time you start a new shell session.
Make sure to replace the placeholders with the appropriate values for your environment.
## Logging
The script will log all deleted resources to a file called `crc.log` in the same directory as the script. The log file will contain the resource type, name, and the date and time it was deleted.
# Usage
To run the script, use the following command:
```
python crc.py --cloud --operation_type --resource --filter_tags --exception_tags --notags --age --slack_channel --influxdb
```
* `cloud`: Specify the cloud name (aws, azure, gcp or all). Required.
* `project_id`: Required for gcp
* `resource`: Indicate the type of resource you want to target (e.g. vm, disk, ip, keypair, spot_instance_requests) or specify "all" to target all types of resources. Default: 'all'
* `resource_group`: Use this option to specify the resource group for Azure. If this flag is provided, the script will only operate on resources within the specified resource group. This allows for more precise targeting of resources in your Azure environment.
* `operation_type`: Specify the type of operation to perform on the resource (delete or stop). Default: 'delete'
* `dry_run`: Enabling this option will only list resources that match the specified criteria without performing any operations on them. Use the `-d` or `--dry_run` flag to enable this feature. If this option is not specified, the script will perform the operation specified by the `operation_type` argument.
* `resource_states`: Specify the state of the resource you want to delete. Only applicable for virtual machines (VMs) and can be either 'RUNNING' or 'STOPPED'. Default: ['RUNNING']. This means that by default, only running VMs will be considered for deletion.
* `filter_tags`: Use this option to filter resources based on their tags. Leave value of Key empty to indicate `any` value. If not specified all available resources will be picked. **This option does not apply to AWS keypairs and GCP IPs**. esources will be included if `any` of the key with `any` value pair matches. (e.g. {'test_task': ['test', 'stress-test']}).
* `exception_tags`: Use this option to exclude resources based on their tags. **Does not apply if `filter_tags` is not set**. Leave the value of Key empty to indicate `any` value. Resources will be excluded if `any` of the key with `any` value pair matches. **This option does not apply to AWS keypairs and GCP IPs** (e.g. {'test_task': ['test-keep-resources', 'stress-test-keep-resources']}).
* `name_regex`: Use this option to filter resources based on regular expressions applied to their names. If not specified, all available resources will be picked. **This option only applies to AWS keypairs and GCP IPs**. Resources will be included if `any` of the specified regular expressions match their names. (e.g. ['perftest_', 'feature_']).
* `exception_regex`: Use this option to exclude resources based on regular expressions applied to their names. **This option does not apply if `name_regex` is not set**. Resources will be excluded if `any` of the specified regular expressions match their names. **This option only applies to AWS keypairs and GCP IPs** (e.g. ['perftest_keep_resources', 'feature_keep_resources'])
* `age`: Use this option to specify an age threshold for resources when deleting resources other than `IPs` (e.g. {'days': 3, 'hours': 12}).
* `custom_age_tag_key`: Define a custom tag key for the age threshold of resources on your cloud resource. This tag is ignored for `IPs` and `Keypairs` (e.g. value of the tag {'days': 3, 'hours': 12}).
* `max_age`: Specify the maximum age threshold for resources. This value will override the value of the tag specified by `custom_age_tag_key` (e.g. {'days': 14, 'hours': 12}).
* `notags`: Use this option to filter resources based on tags that are not present. Leave the value of Key empty to indicate `any` value. Resources will be excluded if `all` of the key-value pair match. This option can be used independently of the `filter_tags` option. **This option does not apply to AWS keypairs and GCP IPs**. Format: -t or --notags {'test_task': ['test'], 'test_owner': []}
* `slack_channel`: Use this option to specify a Slack channel to receive notifications about the execution of the script. Only works if specified.
* `detach_age`: Use this option to specify the detached age for filtering GCP Disks. This option only works for GCP disks deletions.
* `slack_notify_users`: Use this option to tag the user in the Slack notification. It is mandatory to pass `--slack_user_label` with this flag. This option only works for GCP disks deletions.
* `slack_user_label`: Use this option to specify the label to look up in the GCP disks. This option only works for GCP disks deletions.
* `influxdb`: Use this option to specify InfluxDB connection details. The argument takes a dictionary value, with keys 'url', 'org', 'bucket', and an optional key 'resource_suffix'. Example usage: -i or --influxdb {'url': 'http://localhost:8086', 'org': 'Test', 'bucket': 'CRC', 'resource_suffix': 'test'}. Only works if specified.
* `kms_pending_window`: Use this option to specify the number of days before the key actually gets deleted. The number of days
must be between 7 to 30 inclusive.
* `kms_key_description`: Use this option to match specfic string in kms key description.
* `kms_user`: Use this option to specify AWS ARN of user for which keys will be deleted.
# Examples
1. Delete all running AWS VMs that are older than `3 days and 12 hours` and have the tag `test_task` with the value `stress-test`. Additionally, consider the custom age specified in the `retention_age` tag, ensuring the value does not exceed the `max_age` of `14 days`.
```
python crc.py --cloud aws --resource vm --filter_tags "{'test_task': ['stress-test']}" --age "{'days': 3, 'hours': 12}" --custom_age_tag_key retention_age --max_age {'days': 14}
```
2. To stop all Azure VMs in `test-rg` resource group that are older than 2 days and have the tag `test_task` with the value `stress-test`:
```
python crc.py --cloud azure --resource_group test-rg --resource vm --filter_tags "{'test_task': ['stress-test']}" --age "{'days': 2}" --operation_type stop
```
3. To delete all GCP disks that are older than 14 days and have the tag `test_task` with the value `stress-test` and project_id as 'test_project':
```
python crc.py --cloud gcp --project_id test_project --resource disk --filter_tags "{'test_task': ['stress-test']}" --detach_age "{'days': 14}"
```
4. To stop all VMs across all clouds that have the tag `test_task` with the value `stress-test` and `perf-test` and do not have the tag `test_owner`:
```
python crc.py --cloud all --resource vm --filter_tags "{'test_task': ['stress-test', 'perf-test']}" --notags "{'test_owner': []}" --operation_type stop
```
5. To perform a dry run of the script and list all VMs across all clouds that have been created in the last 2 days and do not have the tag `test_task`:
```
python crc.py --cloud all --resource vm --age "{'days': 2}" --notags "{'test_task': []}" --dry_run
```
6. To Delete AWS VPCs and all attached resources
```
python crc.py --cloud aws --resource vm --filter_tags "{'test_task': ['stress-test', 'perf-test']}" --notags "{'test_owner': []}"
```
7. To Delete AWS KMS that are older than 3 days
```
python3 crc.py --cloud aws --resource kms --kms_key_description --kms_user --age "{'days': 3}"
```
8. To delete all AWS Spot Instance Requests & associated Instances that are older than 3 days and 12 hours and have the tag `test_task` with the value `stress-test`:
```
python crc.py --cloud aws --resource spot_instance_requests --filter_tags "{'test_task': ['stress-test']}" --age "{'days': 3, 'hours': 12}"
```
# Notes
* Please make sure to test this script in a non-production environment before using it in a production environment. This script will delete resources permanently and cannot be undone.
* Try using the dry run mode feature to avoid unfortunate circumstances.
* If filters are not specified, the tool will consider every resource for cleanup.
* Use the `filter_tags`, `exception_tags` and `notags` options in JSON format (`Dict[str, List[str]]`)
* Use the `resource_states`, `name_regex`, and `exception_regex` options in list format (`List[str]`)
* When giving a value to the `resource_states` parameter, be aware that different cloud libraries have different formats. (For eg. `running` state for AWS, AZU but `RUNNING` for GCP)
* Use the `age` and `influxdb` option in JSON format. Example: `{"days": 60}` (`Dict[str, int]`)
* VPCs support only `Delete` operation and do not respect `age` threshold.
# Need Help?
* You can ask questions, find answers, and help others on our Community [Slack](https://communityinviter.com/apps/yugabyte-db/register), [Forum](https://forum.yugabyte.com), [Stack Overflow](https://stackoverflow.com/questions/tagged/yugabyte-db), as well as Twitter [@Yugabyte](https://twitter.com/yugabyte)
* Please use [GitHub issues](https://github.com/yugabyte/cloud-resource-cleanup/issues) to report issues or request new features.
# Contribute
As an an open-source project with a strong focus on the user community, we welcome contributions as GitHub pull requests. See our [Contributor Guides](https://docs.yugabyte.com/preview/contribute/) to get going. Discussions and RFCs for features happen on the design discussions section of our [Forum](https://forum.yugabyte.com).
# License
Source code in this repository is licensed under the Apache License 2.0. A copy of license can be found in the [LICENSE.md](LICENSE.md) file.
# Read More
* To see our updates, go to [The Distributed SQL Blog](https://blog.yugabyte.com/).
* For an in-depth design and the YugabyteDB architecture, see our [design specs](https://github.com/yugabyte/yugabyte-db/tree/master/architecture/design).
* Tech Talks and [Videos](https://www.youtube.com/c/YugaByte).
* See how YugabyteDB [compares with other databases](https://docs.yugabyte.com/preview/faq/comparisons/).