https://github.com/s0md3v/Photon

Incredibly fast crawler designed for OSINT.
https://github.com/s0md3v/Photon

crawler information-gathering osint python spider

Last synced: 8 months ago
JSON representation

Incredibly fast crawler designed for OSINT.

• Host: GitHub
• URL: https://github.com/s0md3v/Photon
• Owner: s0md3v
• License: gpl-3.0
• Created: 2018-03-30T19:38:22.000Z (over 7 years ago)
• Default Branch: master
• Last Pushed: 2024-08-16T05:39:15.000Z (about 1 year ago)
• Last Synced: 2025-03-18T04:26:27.625Z (8 months ago)
• Topics: crawler, information-gathering, osint, python, spider
• Language: Python
• Homepage:
• Size: 354 KB
• Stars: 11,404
• Watchers: 322
• Forks: 1,544
• Open Issues: 53
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • License: LICENSE.md

Awesome Lists containing this project

• cybersources - Photon
• Starred-Repo - Photon: Python Web Crawler
• awesome-starred-test - s0md3v/Photon - Incredibly fast crawler designed for OSINT. (Python)
• awesome-rainmana - s0md3v/Photon - Incredibly fast crawler designed for OSINT. (Python)
• awesome-repositories - s0md3v/Photon - Incredibly fast crawler designed for OSINT. (Python)
• WebHackersWeapons - Photon
• my-awesome - Photon - Incredibly fast crawler which extracts urls, emails, files, website accounts and much more. (Tool)
• awesome-thai-osint - Photon - fast Domain OSINT scraper (well-maintained-osint / Meaung, Udon Thani)
• awesome-hacking-lists - s0md3v/Photon - Incredibly fast crawler designed for OSINT. (Python)
• Offensive-OSINT-Tools - Photon - Incredibly fast crawler designed for OSINT. ([↑](#-table-of-contents) SubDomain's / Tools)
• best-of-web-python - GitHub - 43% open · ⏱️ 31.03.2025): (Web Scraping & Crawling)

README

          



  


  

  


  Photon

  




Incredibly fast crawler designed for OSINT.




  

    

  

  

    

  

  

      

  

  

    

  



Met a CAPTCHA? Try [CapSolver](https://www.capsolver.com/?utm_source=github&utm_medium=repo&utm_campaign=scraping&utm_term=photon) solving solution.

![demo](https://image.ibb.co/kQSUcz/demo.png)



  Photon Wiki •

  How To Use •

  Compatibility •

  Photon Library •

  Contribution •

  Roadmap



### Key Features

#### Data Extraction

Photon can extract the following data while crawling:

- URLs (in-scope & out-of-scope)

- URLs with parameters (`example.com/gallery.php?id=2`)

- Intel (emails, social media accounts, amazon buckets etc.)

- Files (pdf, png, xml etc.)

- Secret keys (auth/API keys & hashes)

- JavaScript files & Endpoints present in them

- Strings matching custom regex pattern

- Subdomains & DNS related data

The extracted information is saved in an organized manner or can be [exported as json](https://github.com/s0md3v/Photon/wiki/Usage#export-formatted-result).

![save demo](https://image.ibb.co/dS1BqK/carbon_2.png)

#### Flexible

Control timeout, delay, add seeds, exclude URLs matching a regex pattern and other cool stuff.

The extensive range of [options](https://github.com/s0md3v/Photon/wiki/Usage) provided by Photon lets you crawl the web exactly the way you want.

#### Genius

Photon's smart thread management & refined logic gives you top notch performance.

Still, crawling can be resource intensive but Photon has some tricks up it's sleeves. You can fetch URLs archived by [archive.org](https://archive.org/) to be used as seeds by using `--wayback` option.

#### Plugins

- **[wayback](https://github.com/s0md3v/Photon/wiki/Usage#use-urls-from-archiveorg-as-seeds)**

- **[dnsdumpster](https://github.com/s0md3v/Photon/wiki/Usage#dumping-dns-data)**

- **[Exporter](https://github.com/s0md3v/Photon/wiki/Usage#export-formatted-result)**

#### Docker

Photon can be launched using a lightweight Python-Alpine (103 MB) Docker image.

```bash

$ git clone https://github.com/s0md3v/Photon.git

$ cd Photon

$ docker build -t photon .

$ docker run -it --name photon photon:latest -u google.com

```

To view results, you can either head over to the local docker volume, which you can find by running `docker inspect photon` or by mounting the target loot folder:

```bash

$ docker run -it --name photon -v "$PWD:/Photon/google.com" photon:latest -u google.com

```

#### Frequent & Seamless Updates

Photon is under heavy development and updates for fixing bugs. optimizing performance & new features are being rolled regularly.

If you would like to see features and issues that are being worked on, you can do that on [Development](https://github.com/s0md3v/Photon/projects/1) project board.

Updates can be installed & checked for with the `--update` option. Photon has seamless update capabilities which means you can update Photon without losing any of your saved data.

### Contribution & License

You can contribute in following ways:

- Report bugs

- Develop plugins

- Add more "APIs" for ninja mode

- Give suggestions to make it better

- Fix issues & submit a pull request

Please read the [guidelines](https://github.com/s0md3v/Photon/wiki/Guidelines) before submitting a pull request or issue.

Do you want to have a conversation in private? Hit me up on my [twitter](https://twitter.com/s0md3v/), inbox is open :)

**Photon** is licensed under [GPL v3.0 license](https://www.gnu.org/licenses/gpl-3.0.en.html)