{"id":13542327,"url":"https://github.com/cr0hn/festin","last_synced_at":"2025-04-06T20:10:34.554Z","repository":{"id":57429113,"uuid":"272970751","full_name":"cr0hn/festin","owner":"cr0hn","description":"FestIn - Open S3 Bucket Scanner","archived":false,"fork":false,"pushed_at":"2020-12-04T09:31:35.000Z","size":5783,"stargazers_count":223,"open_issues_count":0,"forks_count":31,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-05-15T12:09:59.445Z","etag":null,"topics":["aws-security","s3","s3-bucket","s3-security"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cr0hn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-17T12:40:53.000Z","updated_at":"2024-04-30T11:44:15.000Z","dependencies_parsed_at":"2022-09-09T00:41:42.399Z","dependency_job_id":null,"html_url":"https://github.com/cr0hn/festin","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cr0hn%2Ffestin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cr0hn%2Ffestin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cr0hn%2Ffestin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cr0hn%2Ffestin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cr0hn","download_url":"https://codeload.github.com/cr0hn/festin/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247543591,"owners_count":20955865,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-security","s3","s3-bucket","s3-security"],"created_at":"2024-08-01T10:01:04.962Z","updated_at":"2025-04-06T20:10:34.532Z","avatar_url":"https://github.com/cr0hn.png","language":"Python","funding_links":[],"categories":["Python","Miscellaneous","SaaS","Python (1887)"],"sub_categories":["Buckets"],"readme":"![Festin logo](https://raw.githubusercontent.com/cr0hn/festin/master/images/festin-logo-banner.png)\n\n## `FestIN` the powered S3 bucket finder and content discover\n\n\u003c!-- START doctoc generated TOC please keep comment here to allow auto update --\u003e\n\u003c!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --\u003e\n\n\n- [What is FestIn](#what-is-festin)\n- [Why Festin](#why-festin)\n- [Install](#install)\n  - [Using Python](#using-python)\n  - [Using Docker](#using-docker)\n- [Full options](#full-options)\n- [Usage](#usage)\n  - [Configure search domains](#configure-search-domains)\n  - [Concurrency](#concurrency)\n  - [HTTP Crawling configuration](#http-crawling-configuration)\n  - [Manage results](#manage-results)\n  - [Proxy usage](#proxy-usage)\n  - [DNS Options](#dns-options)\n  - [Full Text Support](#full-text-support)\n  - [Running as a service (or watching mode)](#running-as-a-service-or-watching-mode)\n- [Example: Mixing FesIn + DnsRecon](#example-mixing-fesin--dnsrecon)\n  - [Step 1 - Run dnsrecon with desired options against target domain and save the output](#step-1---run-dnsrecon-with-desired-options-against-target-domain-and-save-the-output)\n  - [Step 2 - Prepare the previous generated file to feed `FestIn`](#step-2---prepare-the-previous-generated-file-to-feed-festin)\n  - [Step 3 - Run FestIn with desired options and save output](#step-3---run-festin-with-desired-options-and-save-output)\n        - [Run against *target.com* using tor proxy, with concurrency of 5, using DNS 212.166.64.1 for resolving CNAMEs and leaving result to target.com.result file:](#run-against-targetcom-using-tor-proxy-with-concurrency-of-5-using-dns-212166641-for-resolving-cnames-and-leaving-result-to-targetcomresult-file)\n- [F.A.Q.](#faq)\n- [Who uses FestIn](#who-uses-festin)\n- [MrLooquer](#mrlooquer)\n- [License](#license)\n\n\u003c!-- END doctoc generated TOC please keep comment here to allow auto update --\u003e\n\n## What is FestIn\n\n`FestIn` is a tool for discovering open S3 Buckets starting from a domains.\n\nIt perform a lot of test and collects information from:\n\n- DNS\n- Web Pages (Crawler)\n- S3 bucket itself (like S3 redirections)\n\n## Why Festin\n\nThere's a lot of S3 tools for enumeration and discover S3 bucket. Some of them are great but anyone have a complete list of features that `Festin` has. \n\nMain features that does `Festin` great:\n\n- **Various techniques** for finding buckets: crawling, dns crawling and S3 responses analysis.\n- **Proxy** support for tunneling requests.\n- AWS **credentials** are **not needed**.\n- Works with **any S3 compatible** provider, not only with AWS.\n- Allows to configure **custom DNS servers**.\n- Integrated high performance **HTTP crawler**.\n- **Recursively** search and feedback from the 3 engines: a domain found by dns crawler is send to S3 and Http Crawlers analyzer and the same for the S3 and Crawler.\n- Works as **'watching' mode**, listening for new domains in real time.\n- Save all of the domains discovered in a separate file for further analysis.\n- Allow to **download bucket objects** and put then in a **FullText Search Engine** (Redis Search) automatically, indexing the objects content allowing powerful search further.\n- **Limit** the search for specific domain/s.\n\n## Install\n\n### Using Python\n\n    Python 3.8 of above needed!\n\n```bash\n$ pip install festin\n$ festin -h\n```\n\n### Using Docker\n\n```bash\n$ docker run --rm -it cr0hn/festin -h\n```\n\n## Full options\n\n```bash\n$ festin -h\nusage: __main__.py [-h] [--version] [-f FILE_DOMAINS] [-w] [-c CONCURRENCY] [--no-links] [-T HTTP_TIMEOUT] [-M HTTP_MAX_RECURSION] [-dr DOMAIN_REGEX] [-rr RESULT_FILE] [-rd DISCOVERED_DOMAINS] [-ra RAW_DISCOVERED_DOMAINS]\n                   [--tor] [--debug] [--no-print] [-q] [--index] [--index-server INDEX_SERVER] [-dn] [-ds DNS_RESOLVER]\n                   [domains [domains ...]]\n\nFestin - the powered S3 bucket finder and content discover\n\npositional arguments:\n  domains\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --version             show version\n  -f FILE_DOMAINS, --file-domains FILE_DOMAINS\n                        file with domains\n  -w, --watch           watch for new domains in file domains '-f' option\n  -c CONCURRENCY, --concurrency CONCURRENCY\n                        max concurrency\n\nHTTP Probes:\n  --no-links            extract web site links\n  -T HTTP_TIMEOUT, --http-timeout HTTP_TIMEOUT\n                        set timeout for http connections\n  -M HTTP_MAX_RECURSION, --http-max-recursion HTTP_MAX_RECURSION\n                        maximum recursison when follow links\n  -dr DOMAIN_REGEX, --domain-regex DOMAIN_REGEX\n                        only follow domains that matches this regex\n\nResults:\n  -rr RESULT_FILE, --result-file RESULT_FILE\n                        results file\n  -rd DISCOVERED_DOMAINS, --discovered-domains DISCOVERED_DOMAINS\n                        file name for storing new discovered after apply filters\n  -ra RAW_DISCOVERED_DOMAINS, --raw-discovered-domains RAW_DISCOVERED_DOMAINS\n                        file name for storing any domain without filters\n\nConnectivity:\n  --tor                 Use Tor as proxy\n\nDisplay options:\n  --debug               enable debug mode\n  --no-print            doesn't print results in screen\n  -q, --quiet           Use quiet mode\n\nRedis Search:\n  --index               Download and index documents into Redis\n  --index-server INDEX_SERVER\n                        Redis Search ServerDefault: redis://localhost:6379\n\nDNS options:\n  -dn, --no-dnsdiscover\n                        not follow dns cnames\n  -ds DNS_RESOLVER, --dns-resolver DNS_RESOLVER\n                        comma separated custom domain name servers\n```\n\n## Usage\n\n### Configure search domains\n\nBy default `FestIn` accepts a start domain as command line parameter:\n\n```bash\n\u003e festin mydomain.com\n```\n\nBut you also cat setup an external file with a list of domains:\n\n```bash\n\u003e cat domains.txt\ndomain1.com\ndomain2.com\ndomain3.com\n\u003e festin -f domains.txt \n```\n\n### Concurrency\n\n`FestIn` performs a lot of test for a domain. Each test was made concurrently. By default concurrency is set to *5*. If you want to increase the number of concurrency tests you must set the option `-c`\n\n```bash\n\u003e festin -c 10 mydomain.com \n```\n\n    Be carefull with the number of concurrency test or \"alarms\" could raises in some web sites.\n\n### HTTP Crawling configuration\n\n`FestIn` embed a small crawler to discover links to S3 buckets. Crawler accepts these options:\n\n- Timeout (`-T` or `--http-timeout`): configure a timeout for HTTP connections. If website of the domain you want to analyze is slow, we recommend to increase this value. By default timeout is **5 seconds**.\n- Maximum recursion (`-H` or `--http-max-recursion`): this value setup a limit for crawling recursion. Otherwise `FestIn` will scan all internet. By default this value is 3. It means that only will follow: domain1.com -\u003e [link] -\u003e domain2.com -\u003e [link] -\u003e domain3.com -\u003e [link] -\u003e Maximum recursion reached. Stop\n- Limit domains (`-dr` or `--domain-regex`): set this option to limit crawler to these domains that matches with this regex.\n- Black list (-B): configure a black list words file. Each domain that matches with some word in the black list will be skipped.\n- White list (-W): configure a white list words file. Each domain that DOESN'T match with some word in the white list will be skipped.\n\nExample:\n\n```bash\n\u003e echo \"cdn\" \u003e blacklist.txt\n\u003e echo \"photos\" \u003e\u003e blacklist.txt\n\u003e festin -T 20 -M 8 -B blacklist.txt -dr .mydomain. mydomain.com \n```\n\n    BE CAREFUL: -dr (or --domain-regex) only accept valid POSIX regex. \n    \n    *mydomain.com* -\u003e is not a valida POSIX regex\n    .mydomain\\.com. -\u003e is a valida POSIX regex\n\n### Manage results\n\nWhen `FestIn` runs it discover a lot of useful information. Not only about S3 buckets, also for other probes we could do. For example: \n\nAfter we use `FestIn` we can use discovered information (domains, links, resources, other buckets...) as input of other tools, like **nmap**.  \n\nFor above reason `FestIn` has 3 different modes to store discovered information and we can combine them:\n\n- `FestIn` result file (`-rr` or `--result-file`): this file contains one JSON per line with buckets found by them. Each JSON includes: origin domain, bucket name and the list of objects for the bucket.\n- Filtered discovered domains file (`-rd` or `--discovered-domains`): this file contains one domain per line. These domains are discovered by the crawler, dns or S3 probes but only are stored these domains that matches with user and internal filters.\n- Raw discovered domains file (`-ra` or `--raw-discovered-domains` ): this file contains all domains, one per line, discovered by `FestIn` without any filter. This option is useful for post-processing and analyzing.\n\nExample:\n\n```bash\n\u003e festin -rr festin.results -rd discovered-domains.txt -ra raw-domains.txt mydomain.txt \n```\n\nAnd, chaining with Nmap:\n\n```bash\n\u003e festin -rd domains.txt \u0026\u0026 nmap -Pn -A -iL domains.txt -oN nmap-domains.txt \n```\n\n### Proxy usage\n\n`FestIn` embeds the option `--tor`. By using this parameter you need local Tor proxy running at port *9050* at *127.0.0.1*.\n\n```bash\n\u003e tor \u0026\n\u003e festin --tor mydomain.com \n```\n\n### DNS Options\n\nSome tests made by `FestIn` involves DNS. It support these options:\n\n- Disable DNS discovery (`-dn` or `--no-dnsdiscover`)\n- Custom DNS server (`-ds` or `--dns-resolver`): setup custom DNS server. If you plan to perform a lot of tests you should use a different DNS server like you use to your browser.\n\nExample:\n\n```bash\n\u003e festin -ds 8.8.8.8 mydomain.com \n```\n\n### Full Text Support\n\n`FestIn` not only can discover open S3 buckets. It also can download all content and store them in a Full Text Search Engine. **This means that you can perform Full Text Queries to the content of the bucket!**\n\n`FestIn` uses as Full Text Engine the Open Source project [Redis Search](https://oss.redislabs.com/redisearch/).\n\nThis feature has two options:\n\n- Enable indexing (`--index`): to enable the indexing to the search engine you must setup this flag.\n- Redis Search config (`--index-server`): you only need to setup this option if your server is running in a different IP/Port that: *localhost:6379*.\n\nExample:\n\n```bash\n\u003e docker run --rm -p 6700:6379 redislabs/redisearch:latest -d\n\u003e festin --index --index-server redis://127.0.0.1:6700 mydomain.com\n```\n\n    Pay attention to option `--index-server` is must has the prefix **redis://** \n\n\n### Running as a service (or watching mode)\n\nSome times we don't want to stop `FestIn` and launch them some times when we have a new domain to inspect or any external tool discovered new domains we want to check.\n\n`FestIn` supports *watching* mode. This means that `FestIn` will start and listen for new domains. The way to \"send\" new domains to `FestIn` is by domains file. It monitor this file for changes.\n\nThis feature is useful to combine `FestIn` with other tools, like *dnsrecon*\n\nExample:\n\n```bash\n\u003e festin --watch -f domains.txt \n```\n\nIn a different terminal we can write:\n\n```bash\n\u003e echo \"my-second-domain.com\" \u003e\u003e domains.txt \n\u003e echo \"another-domain.com\" \u003e\u003e domains.txt \n```\n\nEach new domain added to *domains.txt* will wakeup `FestIn`.\n\n## Example: Mixing FesTin + DnsRecon\n\n**Using DnsRecon**\n\nThe domain chosen for this example is *target.com*.\n\n### Step 1 - Run dnsrecon with desired options against target domain and save the output\n\n```bash\n\u003e  dnsrecon -d target.com -t crt -c target.com.csv\n```\n\nWith this command we are going to find out other domains related to target.com. This will help to maximize our chances of success.\n\n### Step 2 - Prepare the previous generated file to feed `FestIn` \n\n```bash\n\u003e tail -n +2 target.com.csv | sort -u | cut -d \",\" -f 2 \u003e\u003e target.com.domains\n```\n\nWith this command we generate a file with one domain per line. This is the input that  `FestIn`  needs.\n\n### Step 3 - Run FestIn with desired options and save output\n\n```bash\n\u003e  festin -f target.com.domains -c 5 -rr target.com.result.json --tor -ds 212.166.64.1 \u003etarget.com.stdout 2\u003etarget.com.stderr\n```\n\nIn this example the resulting files are:\n\n- target.com.result.json - Main result file with one line per bucket found. Each line is a JSON object.\n- target.com.stdout - The standard output of festin command execution\n- target.com.stderr - The standard error of festin command execution\n\nIn order to easy the processing of multiple domains, we provide a simple script [examples/loop.sh](https://raw.githubusercontent.com/cr0hn/festin/master/examples/loop.sh) that automatize this.\n\n**Using FestIn with DnsRecon results**\n\nRun against *target.com* with default options and leaving result to target.com.result file:\n\n```bash\n\u003e festin target.com -rr target.com.result.json \n```\n\n###### Run against *target.com* using tor proxy, with concurrency of 5, using DNS 212.166.64.1 for resolving CNAMEs and leaving result to target.com.result file:\n\n```bash\n\u003e festin target.com -c 5 -rr target.com.result.json --tor -ds 212.166.64.1 \n```\n\n## F.A.Q.\n\nQ: AWS bans my IP\nA:\n\nWhen you perform a lot of test against AWS S3, AWS includes your IP in a black list. Then each time you want to access to **any** S3 bucket with `FestIn` of with your browser **will be blocked**.\n\nWe recommend to setup a proxy when you use `FestIn`.    \n\n## Who uses FestIn\n\n## MrLooquer\n\n![Mr looquer](https://raw.githubusercontent.com/cr0hn/festin/master/images/whouses/mrlooquer.jpg)\n\nThey analyze and assess your company risk exposure in real time. [Website](https://www.mrlooquer.com)\n\n## License\n\nThis project is distributed under [BSD license](https://github.com/cr0hn/festin/blob/master/LICENSE\u003e)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcr0hn%2Ffestin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcr0hn%2Ffestin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcr0hn%2Ffestin/lists"}