https://github.com/ucopacme/orgcrawler
A toolset for managing AWS resources across an organization
https://github.com/ucopacme/orgcrawler
aws boto3 organizations
Last synced: 6 months ago
JSON representation
A toolset for managing AWS resources across an organization
- Host: GitHub
- URL: https://github.com/ucopacme/orgcrawler
- Owner: ucopacme
- License: gpl-3.0
- Created: 2018-07-04T16:04:16.000Z (almost 8 years ago)
- Default Branch: master
- Last Pushed: 2020-04-03T01:54:40.000Z (over 6 years ago)
- Last Synced: 2025-09-25T09:31:54.297Z (9 months ago)
- Topics: aws, boto3, organizations
- Language: Python
- Homepage: https://orgcrawler.readthedocs.io/en/latest/
- Size: 197 KB
- Stars: 4
- Watchers: 3
- Forks: 4
- Open Issues: 15
-
Metadata Files:
- Readme: README.rst
- Contributing: CONTRIBUTING.rst
- License: LICENSE
Awesome Lists containing this project
README
Getting Started with OrgCrawler
===============================
A python library for managing resources across all accounts in an AWS Organization.
OrgCrawler package exports two primary classes:
``orgcrawler.orgs.Org``
provides a data model and methods for querying AWS Organizations resources:
- accounts
- organizational units
- service control policies
``orgcrawler.crawlers.Crawler``
provides a framework for executing user defined python code in all accounts and regions or a subset thereof.
OrgCrawler also contains two commandline scripts:
``orgquery``
Provides a generic interface for running organization queries
``orgcrawler``
Execute a custom boto3 payload function in all specified accounts/regions
See full documentation at https://orgcrawler.readthedocs.io/en/latest/
Currently orgcrawler is tested in python 3.6, 3.7.
The OrgCrawler package no longer contains payload functions. These are now maintained
in package ``orgcrawler-payload``.
Installation
------------
::
pip install orgcrawler
pip install orgcrawler-payload
The Org object
--------------
Usage
*****
Create an orgcrawler.orgs.Org instance::
master_account_id = orgcrawler.utils.get_master_account_id('myOrgMasterRole')
my_org = orgcrawler.orgs.Org(master_account_id, 'myOrgMasterRole')
Load your organization's account and organizational unit resources into your instance::
my_org.load()
Query your organization's account and organizational unit resources::
all_accounts = my_org.list_accounts_by_name()
all_org_units = my_org.list_org_units()
test_accounts = my_org.list_accounts_in_ou('testing')
Details
*******
A newly initialized Org object only contains master account information.
The ``load()`` method makes boto3 client calls to the master account to
populate the Org object's Account and OrganizationalUnit resource data.
The `loaded` Org object is cached to local disk to improve the performance of
subsequent ``load()`` calls. After an hour, the cached Org object is timed
out. It contains no AWS account credentials.
Org object query methods can return listings of accounts based on OrganizationalUnit membership.
See the API docs for a full listing of query methods
The Crawler object
------------------
The orgcrawler.crawlers.Crawler object provides a framework for running python
functions in all accounts and regions in your organization or a subset thereof.
Create an orgcrawler.crawlers.Crawler instance::
my_crawler = orgcrawler.crawlers.Crawler(
my_org,
access_role='myAccountAdminRole',
accounts=test_accounts,
regions=['us-west-1', 'us-east-1'],
)
Load AWS credentials for each account into your Crawler instance::
my_crawler.load_account_credentials()
Shortcut Crawler Setup
----------------------
You can generate an Crawler object and associated Org object with a single utility function::
from orgcrawler.cli.utils import setup_crawler
my_crawler = crawler_setup('MyMasterAccountRole')
or::
crawler_params = dict(
access_role='myAccountAdminRole',
accounts=test_accounts,
regions=['us-west-1', 'us-east-1'],
)
my_crawler = crawler_setup('MyMasterAccountRole', **crawler_params)
Running Crawler Payload Functions
---------------------------------
Run your custom python code in all account/regions configured in your Crawler
by calling the ``execute()`` method of your Crawler instance. Supply your
`payload` function and any function arguments as parameters::
import my_payloads
all_buckets = my_crawler.execute(my_payloads.list_s3_buckets)
my_crawler.execute(my_payloads.create_bucket, 'my_bucket')
Requirments for Payload Functions
---------------------------------
The Crawler.execute method calls your custom function with the following
parameters: ``region, account, *args``, where ``region`` is a string,
``account`` is orgcrawler.orgs.OrgAccount instance, and ``args`` is a list of
positional parameters to pass to your function. Your function must create its
own boto3 client for whatever services it will use.
Examples::
def list_s3_buckets(region, account):
client = boto3.client('s3', region_name=region, **account.credentials)
response = client.list_buckets()
def create_bucket(region, account, bucket_prefix):
client = boto3.client('s3', region_name=region, **account.credentials)
bucket_name = '-'.join([bucket_prefix, account.id, region])
bucket_attributes = {'Bucket': bucket_name}
if not region == 'us-east-1':
bucket_attributes['CreateBucketConfiguration'] = {'LocationConstraint': region}
response = client.create_bucket(**bucket_attributes)
Running payloads which call global AWS services
-----------------------------------------------
When calling a payload function which accesses a global service, set the
``regions`` attribute of your Crawler instance either to ``us-east-1`` or to
the keywork ``GLOBAL``. Otherwise, Crawler.execute() will travers every AWS
region and redandantly run your payload::
my_crawler = crawler_setup('MyMasterAccountRole')
my_crawler.update_regions('GLOBAL')
my_crawler.execute(iam_list_users)
OrgCrawler CLI Scripts
----------------------
This package contains two console scripts: ``orgquery`` and ``orgcrawler``.
These attempt to provide a generic interface for running organization queries
and custom crawler functions. They provide concrete examples for how to build
orgcrawler applications.
See ``orgcrawler/cli/{orgquery|orgcrawler}.py`` for code.
Run with the ``--help`` option for usage.
CLI Examples
------------
orgquery
********
::
orgquery -h
orgquery -r OrgMasterRole list_accounts_by_name
orgquery -r OrgMasterRole -f yaml get_account_id_by_name webapps
orgquery -r OrgMasterRole get_targets_for_policy LimitAWSRegions | jq -re .[].Name
orgcrawler
**********
::
orgcrawler -h
orgcrawler -r OrgMasterRole orgcrawler.payloads.list_buckets
orgcrawler -r OrgMasterRole --account-role S3Admin orgcrawler.payloads.create_bucket orgcrawler-testbucket
orgcrawler -r OrgMasterRole --service codecommit --payload-file ~/my_payloads.py list_cc_repositories
orgcrawler -r OrgMasterRole --accounts app-test,app-prod --regions us-east-1,us-west-2 orgcrawler.payloads.config_describe_rules
When running a payload function which calls a global AWS service such as IAM or
Route53, use the ``--service`` option. This will set the ``regions`` attibute
of your crawler object to the default region ``us-east-1``::
orgcrawler -r OrgMasterRole --service iam orgcrawler.payloads.iam_list_users
orgcrawler -r OrgMasterRole --service route53 orgcrawler.payloads.list_hosted_zones