{"id":13821199,"url":"https://github.com/flosell/trailscraper","last_synced_at":"2025-04-08T09:07:32.889Z","repository":{"id":26984080,"uuid":"112079872","full_name":"flosell/trailscraper","owner":"flosell","description":"A command-line tool to get valuable information out of AWS CloudTrail","archived":false,"fork":false,"pushed_at":"2024-04-08T01:29:33.000Z","size":680,"stargazers_count":761,"open_issues_count":22,"forks_count":35,"subscribers_count":13,"default_branch":"master","last_synced_at":"2024-04-13T03:26:54.484Z","etag":null,"topics":["aws","aws-cloudformation","aws-cloudtrail","cloud","cloudtrail","cloudtrail-consumer","cloudtrail-log-analytics","hacktoberfest","heuristic","iam","iam-actions","security","security-automation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flosell.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2017-11-26T12:15:52.000Z","updated_at":"2024-04-15T02:39:20.149Z","dependencies_parsed_at":"2023-11-06T02:37:22.184Z","dependency_job_id":"4545f302-32ec-4eb6-921a-e0eb5f918c50","html_url":"https://github.com/flosell/trailscraper","commit_stats":{"total_commits":940,"total_committers":6,"mean_commits":"156.66666666666666","dds":0.5191489361702128,"last_synced_commit":"6568c870fa6a2785edbd9485f4664b8e140dac52"},"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flosell%2Ftrailscraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flosell%2Ftrailscraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flosell%2Ftrailscraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flosell%2Ftrailscraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flosell","download_url":"https://codeload.github.com/flosell/trailscraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247809962,"owners_count":20999816,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-cloudformation","aws-cloudtrail","cloud","cloudtrail","cloudtrail-consumer","cloudtrail-log-analytics","hacktoberfest","heuristic","iam","iam-actions","security","security-automation"],"created_at":"2024-08-04T08:01:17.357Z","updated_at":"2025-04-08T09:07:32.869Z","avatar_url":"https://github.com/flosell.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# TrailScraper\n\n[![PyPi Release](https://img.shields.io/pypi/v/trailscraper.svg)](https://pypi.python.org/pypi/trailscraper)\n[![Build Status](https://github.com/flosell/trailscraper/actions/workflows/check.yml/badge.svg)](https://github.com/flosell/trailscraper/actions/workflows/check.yml)\n\nA command-line tool to get valuable information out of AWS CloudTrail and a general purpose toolbox for working with IAM policies\n\n## Installation\n\n### OSX\n\n```bash\n$ brew install trailscraper\n```\n\n### Installation using pip\n\nRequirements:\n\n* Python \u003e= 3.5\n* pip\n\n```bash\n$ pip install trailscraper\n```\n\n### Run directly using docker\n\n```bash\n$ docker run --rm --env-file \u003c(env | grep AWS_) -v $HOME/.aws:/root/.aws ghcr.io/flosell/trailscraper:latest\n```\n\nCurrent Versions starting from 0.7.0 are found on GitHub Container Registry (`ghcr.io`), older versions on [DockerHub](https://hub.docker.com/r/flosell/trailscraper/)\n\n## Usage\n\n* [Get CloudTrail events matching a filter from CloudTrail API](#get-cloudtrail-events-matching-a-filter-from-cloudtrail-api)\n* [Download some logs](#download-some-logs)\n* [Download some logs in organisational trails](#download-some-logs-in-organisational-trails)\n* [Find CloudTrail events matching a filter in downloaded logs](#find-cloudtrail-events-matching-a-filter-in-downloaded-logs)\n* [Generate Policy from some CloudTrail records](#generate-policy-from-some-cloudtrail-records)\n* [Extend existing policy by guessing matching actions](#extend-existing-policy-by-guessing-matching-actions)\n* [Find CloudTrail events and generate an IAM Policy](#find-cloudtrail-events-and-generate-an-iam-policy)\n\n### Get CloudTrail events matching a filter from CloudTrail API \n\n```\n$ trailscraper select --use-cloudtrail-api \\ \n                      --filter-assumed-role-arn some-arn \\ \n                      --from 'one hour ago' \\ \n                      --to 'now'\n{\n  \"Records\": [\n    {\n      \"eventTime\": \"2017-12-11T15:01:51Z\",\n      \"eventSource\": \"autoscaling.amazonaws.com\",\n      \"eventName\": \"DescribeLaunchConfigurations\",\n...\n```\n\n### Download some logs\n\n```\n$ trailscraper download --bucket some-bucket \\\n                        --account-id some-account-id \\\n                        --region some-other-region \\ \n                        --region us-east-1 \\\n                        --from 'two days ago' \\\n                        --to 'now' \\\n```\n_Note: Include us-east-1 to download logs for global services. See [below](#why-is-trailscraper-missing-some-events) for details_\n\n### Download some logs in organisational trails\n\n```\n$ trailscraper download --bucket some-bucket \\\n                        --account-id some-account-id \\\n                        --region us-east-1 \\\n                        --org-id o-someorgid \\\n                        --from 'two days ago' \\\n                        --to 'now'\n```\n\n### Find CloudTrail events matching a filter in downloaded logs\n\n```\n$ trailscraper select --filter-assumed-role-arn some-arn \\ \n                      --from 'one hour ago' \\ \n                      --to 'now'\n{\n  \"Records\": [\n    {\n      \"eventTime\": \"2017-12-11T15:01:51Z\",\n      \"eventSource\": \"autoscaling.amazonaws.com\",\n      \"eventName\": \"DescribeLaunchConfigurations\",\n...\n```\n\n### Generate Policy from some CloudTrail records\n\n```\n$ gzcat some-records.json.gz | trailscraper generate\n{\n    \"Statement\": [\n        {\n            \"Action\": [\n                \"ec2:DescribeInstances\"\n            ],\n            \"Effect\": \"Allow\",\n            \"Resource\": [\n                \"*\"\n            ]\n        }\n    ],\n    \"Version\": \"2012-10-17\"\n} \n```\n\n### Extend existing policy by guessing matching actions\n\nCloudTrail logs might not always contain all relevant actions. \nFor example, your logs might only contain the `Create` actions after a terraform run when you really want the delete and\nupdate permissions as well. TrailScraper can try to guess additional statements that might be relevant:  \n\n```\n$ cat minimal-policy.json | trailscraper guess\n{\n    \"Statement\": [\n        {\n            \"Action\": [\n                \"s3:PutObject\"\n            ],\n            \"Effect\": \"Allow\",\n            \"Resource\": [\n                \"*\"\n            ]\n        },\n        {\n            \"Action\": [\n                \"s3:DeleteObject\",\n                \"s3:GetObject\",\n                \"s3:ListObjects\"\n            ],\n            \"Effect\": \"Allow\",\n            \"Resource\": [\n                \"*\"\n            ]\n        }\n    ],\n    \"Version\": \"2012-10-17\"\n}\n$ cat minimal-policy.json | ./go trailscraper guess --only Get\n{\n    \"Statement\": [\n        {\n            \"Action\": [\n                \"s3:PutObject\"\n            ],\n            \"Effect\": \"Allow\",\n            \"Resource\": [\n                \"*\"\n            ]\n        },\n        {\n            \"Action\": [\n                \"s3:GetObject\"\n            ],\n            \"Effect\": \"Allow\",\n            \"Resource\": [\n                \"*\"\n            ]\n        }\n    ],\n    \"Version\": \"2012-10-17\"\n}\n```\n\n### Find CloudTrail events and generate an IAM Policy\n```\n$ trailscraper select | trailscraper generate\n{\n    \"Statement\": [\n        {\n            \"Action\": [\n                \"ec2:DescribeInstances\",\n                \"ec2:DescribeSecurityGroups\",\n                \"ec2:DescribeSubnets\",\n                \"ec2:DescribeVolumes\",\n                \"ec2:DescribeVpcs\",\n            ],\n            \"Effect\": \"Allow\",\n            \"Resource\": [\n                \"*\"\n            ]\n        },\n        {\n            \"Action\": [\n                \"sts:AssumeRole\"\n            ],\n            \"Effect\": \"Allow\",\n            \"Resource\": [\n                \"arn:aws:iam::1111111111:role/someRole\"\n            ]\n        }\n    ],\n    \"Version\": \"2012-10-17\"\n} \n```\n\n## FAQ\n\n### How can I generate policies in CloudFormation YAML instead of JSON? \n\nTrailScraper doesn't provide this. But you can use [cfn-flip](https://github.com/awslabs/aws-cfn-template-flip) to do it:\n\n```\n$ trailscraper select | trailscraper generate | cfn-flip\nStatement:\n  - Action:\n      - ec2:DescribeInstances\n    Effect: Allow\n    Resource:\n      - '*'\n```\n\n### How can I generate policies in Terraform HCL instead of JSON? \n\nTrailScraper doesn't provide this. But you can use [iam-policy-json-to-terraform](https://github.com/flosell/iam-policy-json-to-terraform) to do it:\n\n```\n$ trailscraper select | trailscraper generate | iam-policy-json-to-terraform\ndata \"aws_iam_policy_document\" \"policy\" {\n  statement {\n    sid       = \"\"\n    effect    = \"Allow\"\n    resources = [\"*\"]\n\n    actions = [\n      \"ec2:DescribeInstances\",\n    ]\n  }\n}\n```\n\n### Why is TrailScraper missing some events?\n\n* Make sure you have logs for the `us-east-1` region. Some global AWS services (e.g. Route53, IAM, STS, CloudFront) use this region. For details, check the [CloudTrail Documentation](http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-concepts.html#cloudtrail-concepts-global-service-events)\n\n### Why are some TrailScraper-generated actions not real IAM actions?\n\nThis is totally possible. Unfortunately, there is no good, machine-readable documentation on how CloudTrail events\nmap to IAM actions so TrailScraper is using heuristics to figure out the right actions. These heuristics likely don't\ncover all special cases of the AWS world.\n\nThis is where you come in: If you find a special case that's not covered by TrailScraper, \nplease [open a new issue](https://github.com/flosell/trailscraper/issues/new) or, even better, submit a pull request.\n\nFor more details, check out the [contribution guide](./CONTRIBUTING.md) \n\n### Why does click think I am in an ASCII environment? \n\n`Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment.`\n\nSet environment variables that describe your locale, e.g. :\n```\nexport LC_ALL=de_DE.utf-8\nexport LANG=de_DE.utf-8\n```\nor \n```\nLC_ALL=C.UTF-8\nLANG=C.UTF-8\n```\nFor details, see http://click.pocoo.org/5/python3/#python-3-surrogate-handling\n\n## Development\n\n```bash\n$ ./go setup   # set up venv, dependencies and tools\n$ ./go test    # run some tests\n$ ./go check   # run some style checks\n$ ./go         # let's see what we can do here\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflosell%2Ftrailscraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflosell%2Ftrailscraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflosell%2Ftrailscraper/lists"}