https://github.com/usethisname1419/connectioncrawler
crawls a website and checks for connections
https://github.com/usethisname1419/connectioncrawler
connection crawler http-headers reporting website-analyzer
Last synced: 8 months ago
JSON representation
crawls a website and checks for connections
- Host: GitHub
- URL: https://github.com/usethisname1419/connectioncrawler
- Owner: usethisname1419
- Created: 2024-03-01T17:38:44.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-03-01T21:54:52.000Z (almost 2 years ago)
- Last Synced: 2025-01-25T05:38:04.801Z (about 1 year ago)
- Topics: connection, crawler, http-headers, reporting, website-analyzer
- Language: Python
- Homepage:
- Size: 11.7 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Connection Crawler
Site Checker is a Python script that checks a website for outgoing connections, resolves domain names, and generates a report of the findings. It can be used to inspect websites for potentially malicious scripts, connections, or other suspicious activity.
Can be used to find unwanted connections and malware
## Features
- Checks for outgoing connections from the specified website.
- Resolves domain names to their corresponding IP addresses.
- Generates a detailed report containing HTTP response headers, resolved IP addresses, and connections made by the website.
- Supports analysis of HTML tags including , ![]()
, , and <link>.
- Uses BeautifulSoup for HTML parsing and Requests for HTTP requests.
## malicious-connections
integrated connection crawler with urlhaus and added random user-agents to be able to crawl better and check if there is any connections to malicious IPs (requires URLhaus api key)
## Usage
1. Clone the repository:
```
git clone https://github.com/your-username/site-checker.git
```
2. Navigate to the project directory:
```
cd site-checker
```
3. Install the required dependencies:
```
pip install -r requirements.txt
```
4. Run the script:
```
python crawler.py
```
5. Follow the prompts to enter the URL to check and specify the filename for the report.
## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
## Acknowledgements
- [Requests](https://docs.python-requests.org/en/latest/) - HTTP library for making requests in Python.
- [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) - Python library for pulling data out of HTML and XML files.