{"id":20275637,"url":"https://github.com/michenriksen/bucketlist","last_synced_at":"2025-04-11T05:24:28.850Z","repository":{"id":146213606,"uuid":"98293577","full_name":"michenriksen/bucketlist","owner":"michenriksen","description":"Amazon S3 bucket spelunking!","archived":false,"fork":false,"pushed_at":"2017-08-23T13:45:37.000Z","size":17,"stargazers_count":86,"open_issues_count":1,"forks_count":19,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-10T19:32:05.215Z","etag":null,"topics":["osint","ruby","s3","security"],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/michenriksen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-07-25T10:16:18.000Z","updated_at":"2025-03-09T01:07:10.000Z","dependencies_parsed_at":null,"dependency_job_id":"93e3ea1a-d8e9-4639-9a74-8685300c39c6","html_url":"https://github.com/michenriksen/bucketlist","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michenriksen%2Fbucketlist","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michenriksen%2Fbucketlist/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michenriksen%2Fbucketlist/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michenriksen%2Fbucketlist/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/michenriksen","download_url":"https://codeload.github.com/michenriksen/bucketlist/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248346682,"owners_count":21088498,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["osint","ruby","s3","security"],"created_at":"2024-11-14T13:10:23.599Z","updated_at":"2025-04-11T05:24:28.841Z","avatar_url":"https://github.com/michenriksen.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Bucketlist\n\nBucketlist is a quick project I threw together to find and crawl [Amazon S3] buckets and put all the data into a PostgreSQL database for querying.\n\n## Requirements\n\nBucketlist requires a recent version of Ruby and the PostgreSQL database system installed.\n\n## Setup\n\n * Check out the code to a location of your choice and navigate to it in a terminal\n * Install Bundler (unless you already have it) with: `gem install bundler`\n * Install gem dependencies with `bundle install`\n * Create a new PostgreSQL user with: `createuser -s bucketlist --pwprompt` (you might need to `sudo su postgres` first)\n * Create a new PostgreSQL database with: `createdb -O bucketlist bucketlist` (you might need to `sudo su postgres` first)\n * Copy the example configuration file with: `cp config.yml.example config.yml`\n * Edit the settings in `config.yml` to match your setup\n * ???\n * Profit!\n\n## Finding Buckets\n\nBucketlist finds buckets using a dictionary brute force, a bit like subdomain bruteforcing, so you will need a dictionary of words. The [SecLists] project on GitHub has a good collection of wordlists.\n\nWhen you have a wordlist, simply run the `bucket_finder` script in a terminal:\n\n    $ bin/bucket_finder path/to/wordlist.lst\n     - PRIVATE: https://s3.amazonaws.com/somebucket.backups/\n     - PRIVATE: https://s3.amazonaws.com/somebucket.backup/\n     - PRIVATE: https://s3.amazonaws.com/backups.somebucket/\n     - PRIVATE: https://s3.amazonaws.com/backup.somebucket/\n     +  PUBLIC: https://somebucket.dev.s3.amazonaws.com/\n     - PRIVATE: https://s3.amazonaws.com/production.somebucket/\n     ...\n\nThe script will find buckets and store information about them in the database. The script can be stopped at any time. If you run it again with the same wordlist, it will proceed where it left off.\n\n### Bucket name permutations\n\nTo maximize discovery, bucket_finder will perform simple permutations on each word in the given wordlist. As an example, if the wordlist contains the word `example`, bucket_finder will check for the existance of any of following buckets:\n\n```\nexample\nexample.backup\nbackup.example\nexample.backups\nbackups.example\nexample.dev\ndev.example\nexample.development\ndevelopment.example\nexample.prod\nprod.example\nexample.production\nproduction.example\nexample.stage\nstage.example\nexample.staging\nstaging.example\nexample.test\ntest.example\nexample.testing\ntesting.example\nexample-backup\nbackup-example\nexample-backups\nbackups-example\nexample-dev\ndev-example\nexample-development\ndevelopment-example\nexample-prod\nprod-example\nexample-production\nproduction-example\nexample-stage\nstage-example\nexample-staging\nstaging-example\nexample-test\ntest-example\nexample-testing\ntesting-example\nexamplebackup\nbackupexample\nexamplebackups\nbackupsexample\nexampledev\ndevexample\nexampledevelopment\ndevelopmentexample\nexampleprod\nprodexample\nexampleproduction\nproductionexample\nexamplestage\nstageexample\nexamplestaging\nstagingexample\nexampletest\ntestexample\nexampletesting\ntestingexample\n```\n\n## Crawling Buckets\n\nWhen buckets have been discovered with `bucket_finder`, the `bucket_crawler` script can be used to crawl the contents of the public buckets and save information about the files to the database:\n\n    $ bin/bucket_crawler\n    ├── https://somebucket.dev.s3.amazonaws.com/\n    │   ├── PRIVATE: https://somebucket.dev.s3.amazonaws.com/logs/2014-10-11-21-44-41-0DE7B75AC6F56AB6 (276B)\n    │   ├── PRIVATE: https://somebucket.dev.s3.amazonaws.com/logs/2014-10-11-22-17-33-0EF1F7575568BC41 (374B)\n    │   ├── PRIVATE: https://somebucket.dev.s3.amazonaws.com/logs/2014-10-11-21-30-12-9517510CD37C9D98 (320B)\n    ...\n    │   ├── PRIVATE: https://somebucket.dev.s3.amazonaws.com/logs/2014-11-07-09-34-44-A23E12B5C822DEB0 (375B)\n    │   ├── PRIVATE: https://somebucket.dev.s3.amazonaws.com/logs/2014-11-07-10-51-12-4DB562D370986482 (374B)\n    │   ├── PRIVATE: https://somebucket.dev.s3.amazonaws.com/logs/2014-11-07-11-17-56-A58FF2F17296FB3E (375B)\n    ├── https://s3.amazonaws.com/someotherbucket/\n    │   ├──  PUBLIC: https://s3.amazonaws.com/someotherbucket/3-DuisUtRisusCursus.mp4 (9MB)\n    │   ├──  PUBLIC: https://s3.amazonaws.com/someotherbucket/crossdomain.xml (198B)\n    │   ├──  PUBLIC: https://s3.amazonaws.com/someotherbucket/6-AeneanLobortisRutrumLoremEuFermentum.mp4 (19MB)\n    ...\n\n The bucket_crawler script will find any public bucket in the database that hasn't been crawled yet, and can be run at any time.\n\n## Browsing the Loot\n\nAll the data collected by `bucket_finder` and `bucket_crawler` is stored in a simple database schema and can of course be queried in all kinds of interesting ways with SQL, but Bucketlist also includes a simple web application for browsing the information in a convenient way. You can start the web application with:\n\n    $ bin/webapp\n    == Sinatra (v2.0.0) has taken the stage on 3000 for production with backup from Thin\n    Thin web server (v1.7.2 codename Bachmanity)\n    Maximum connections set to 1024\n    Listening on 0.0.0.0:3000, CTRL+C to stop\n\nNow you can browse to [http://localhost:3000/](http://localhost:3000/) and go Bucket spelunking!\n\n## DISCLAIMER\n\nThis code is meant for security professionals. I take **no** responsibility and assume no liability for the manner in which this code is used by you.\n\n[Amazon S3]: https://aws.amazon.com/s3/\n[SecLists]: https://github.com/danielmiessler/SecLists\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichenriksen%2Fbucketlist","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmichenriksen%2Fbucketlist","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichenriksen%2Fbucketlist/lists"}