https://github.com/timursevimli/proxy-scraper-checker
Multifunctional proxy scraper and checker
https://github.com/timursevimli/proxy-scraper-checker
http https proxy proxy-check proxy-checker proxy-list proxy-scraper proxy-scrapper socks4 socks5
Last synced: 9 months ago
JSON representation
Multifunctional proxy scraper and checker
- Host: GitHub
- URL: https://github.com/timursevimli/proxy-scraper-checker
- Owner: timursevimli
- License: mit
- Created: 2022-12-30T14:36:35.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2024-02-15T14:57:56.000Z (almost 2 years ago)
- Last Synced: 2025-04-07T01:22:56.794Z (10 months ago)
- Topics: http, https, proxy, proxy-check, proxy-checker, proxy-list, proxy-scraper, proxy-scrapper, socks4, socks5
- Language: JavaScript
- Homepage:
- Size: 14.3 MB
- Stars: 5
- Watchers: 2
- Forks: 1
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Proxy Scraper
Proxy Scraper is a tool that allows you to check the connectivity to HTTP, HTTPS, SOCKS4, and SOCKS5 proxy servers, as well as collect IP addresses and ports from websites specified in a text file.
## Installation
To install and use Proxy Scraper, follow these steps:
1. Clone the repository using the command:
```bash
git clone https://github.com/timursevimli/proxy-scraper/
```
## Usage
To use Proxy Scraper, follow these steps:
1. Create a text file that lists the websites to collect IP addresses and ports from. Each URL should be specified on a separate line. For example (sources/proxy_sources.txt):
```bash
https://example.com
http://example.net
```
2. Edit the config.json file to configure the script execution and proxy checking options.
```javascript
{
"mode": "single",
"timeout": 10000,
"source": "proxy_sources.txt",
"test": false,
"channels": 100
}
```
3. Run the script by executing the command:
```bash
npm start
```
4. The logs will be recorded in the logs/proxy-scraper.log file. You can view the log file to track the execution and check for results or errors.
## Execution Options
When running the proxy-scaper, you can specify the following options:
- mode< string >(default: 'single'): The type of proxy checking execution. Available values are 'single' for sequential checking and 'multi' for parallel checking.
- timeout< number >(default: 10000): The timeout for each proxy check in milliseconds.
- source< string >(default: 'proxy_sources.txt'): The path to the file containing the websites to collect IP addresses and ports from.
- test< boolean >(default: false): A flag indicating whether to run in test mode (checking only a few websites).
- channels< number >(default: 100) This property limits the number of operations and provides load balance in the application.
## Contribution
You can contribute to the development of Proxy Scraper by creating new features, fixing bugs, or improving the documentation. If you have any suggestions, please feel free to open an issue or submit a pull request.
## License
This project is licensed under the [MIT LICENSE](https://github.com/timursevimli/proxy-scraper/blob/main/LICENSE)