https://github.com/obaskly/broken-links-checker
Check broken links in a website
https://github.com/obaskly/broken-links-checker
404 checker requests selenium
Last synced: 7 months ago
JSON representation
Check broken links in a website
- Host: GitHub
- URL: https://github.com/obaskly/broken-links-checker
- Owner: obaskly
- Created: 2023-08-14T18:08:01.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2023-08-14T19:17:31.000Z (about 2 years ago)
- Last Synced: 2025-01-08T16:30:25.974Z (9 months ago)
- Topics: 404, checker, requests, selenium
- Language: Python
- Homepage:
- Size: 116 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# 🌐 Broken Links Checker
A Python-based tool designed to efficiently identify broken links on a given website. With concurrent requests and the integration of Selenium, ensure that you're providing the best user experience by keeping all your links in check.
## 🚀 Features
- **Proxy Support**: Seamlessly fetches and utilizes proxies for accurate link validation.
- **User-Agent Rotation**: Simulates various devices and browsers.
- **Selenium**: Deep link inspections, especially when handling challenges like Cloudflare.
- **Fast**: Concurrent requests via multi-threading.
## 📋 Prerequisites
- Python (3.6 or higher)
- ChromeDriver (for Selenium)
- Required Python libraries:
```
pip install selenium fake_useragent tqdm colorama cloudscraper requests
```
## 🎈 Usage
1. **Clone and navigate**:
```
git clone https://github.com/obaskly/Broken-Links-Checker.git
cd Broken-Links-Checker
```
2. **Execute the script**:
```
python main.py
```
3. **Follow the on-screen instructions**:
- Submit your website's URL.
- Determine the number of proxies to fetch.
4. **Results**:
- The script will filter working proxies.
- The script will list any broken links discovered.
## 🤝 Contributing
1. Fork the repository.
2. Create a new branch (`git checkout -b feature/AmazingFeature`).
3. Commit your changes (`git commit -m 'Add some AmazingFeature'`).
4. Push to the branch (`git push origin feature/AmazingFeature`).
5. Open a pull request.
## 📜 License
This project is licensed under the MIT License.
## 🌟 Acknowledgements
- [Selenium](https://www.selenium.dev/)
- [fake_useragent](https://pypi.org/project/fake-useragent/)
- [cloudscraper](https://github.com/VeNoMouS/cloudscraper)
- And other libraries used in the project.