Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/s0md3v/Photon

Incredibly fast crawler designed for OSINT.
https://github.com/s0md3v/Photon

crawler information-gathering osint python spider

Last synced: about 1 month ago
JSON representation

Incredibly fast crawler designed for OSINT.

Awesome Lists containing this project

README 

        




  


  Photon

  


  Photon

  





Incredibly fast crawler designed for OSINT.





  

    

  

  

    pypi

  

  

      

  

  

    

  




Met a CAPTCHA? Try [CapSolver](https://www.capsolver.com/?utm_source=github&utm_medium=repo&utm_campaign=scraping&utm_term=photon) solving solution.



![demo](https://image.ibb.co/kQSUcz/demo.png)





  Photon Wiki •

  How To Use •

  Compatibility •

  Photon Library •

  Contribution •

  Roadmap




### Key Features



#### Data Extraction

Photon can extract the following data while crawling:



- URLs (in-scope & out-of-scope)

- URLs with parameters (`example.com/gallery.php?id=2`)

- Intel (emails, social media accounts, amazon buckets etc.)

- Files (pdf, png, xml etc.)

- Secret keys (auth/API keys & hashes)

- JavaScript files & Endpoints present in them

- Strings matching custom regex pattern

- Subdomains & DNS related data



The extracted information is saved in an organized manner or can be [exported as json](https://github.com/s0md3v/Photon/wiki/Usage#export-formatted-result).



![save demo](https://image.ibb.co/dS1BqK/carbon_2.png)



#### Flexible

Control timeout, delay, add seeds, exclude URLs matching a regex pattern and other cool stuff.

The extensive range of [options](https://github.com/s0md3v/Photon/wiki/Usage) provided by Photon lets you crawl the web exactly the way you want.



#### Genius

Photon's smart thread management & refined logic gives you top notch performance.



Still, crawling can be resource intensive but Photon has some tricks up it's sleeves. You can fetch URLs archived by [archive.org](https://archive.org/) to be used as seeds by using `--wayback` option.



#### Plugins

- **[wayback](https://github.com/s0md3v/Photon/wiki/Usage#use-urls-from-archiveorg-as-seeds)**

- **[dnsdumpster](https://github.com/s0md3v/Photon/wiki/Usage#dumping-dns-data)**

- **[Exporter](https://github.com/s0md3v/Photon/wiki/Usage#export-formatted-result)**



#### Docker



Photon can be launched using a lightweight Python-Alpine (103 MB) Docker image.



```bash

$ git clone https://github.com/s0md3v/Photon.git

$ cd Photon

$ docker build -t photon .

$ docker run -it --name photon photon:latest -u google.com

```



To view results, you can either head over to the local docker volume, which you can find by running `docker inspect photon` or by mounting the target loot folder:



```bash

$ docker run -it --name photon -v "$PWD:/Photon/google.com" photon:latest -u google.com

```



#### Frequent & Seamless Updates

Photon is under heavy development and updates for fixing bugs. optimizing performance & new features are being rolled regularly.



If you would like to see features and issues that are being worked on, you can do that on [Development](https://github.com/s0md3v/Photon/projects/1) project board.



Updates can be installed & checked for with the `--update` option. Photon has seamless update capabilities which means you can update Photon without losing any of your saved data.



### Contribution & License

You can contribute in following ways:



- Report bugs

- Develop plugins

- Add more "APIs" for ninja mode

- Give suggestions to make it better

- Fix issues & submit a pull request



Please read the [guidelines](https://github.com/s0md3v/Photon/wiki/Guidelines) before submitting a pull request or issue.



Do you want to have a conversation in private? Hit me up on my [twitter](https://twitter.com/s0md3v/), inbox is open :)



**Photon** is licensed under [GPL v3.0 license](https://www.gnu.org/licenses/gpl-3.0.en.html)