Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/1uc1f3r616/fastproxy

MultiThreaded Application to Scrape Working Web Proxies
https://github.com/1uc1f3r616/fastproxy

beginner-friendly contributions-welcome hacktoberfest hacktoberfest-accepted hide ip live proxy pytest python scrape scraping security threading web

Last synced: 6 days ago
JSON representation

MultiThreaded Application to Scrape Working Web Proxies

Host: GitHub
URL: https://github.com/1uc1f3r616/fastproxy
Owner: 1UC1F3R616
License: mit
Created: 2019-09-04T18:13:03.000Z (over 5 years ago)
Default Branch: master
Last Pushed: 2024-11-17T12:33:28.000Z (about 2 months ago)
Last Synced: 2025-01-02T14:08:40.000Z (14 days ago)
Topics: beginner-friendly, contributions-welcome, hacktoberfest, hacktoberfest-accepted, hide, ip, live, proxy, pytest, python, scrape, scraping, security, threading, web
Language: Python
Homepage: https://pypi.org/project/fastProxy/
Size: 119 KB
Stars: 33
Watchers: 3
Forks: 10
Open Issues: 0
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

        


  





  MultiThreaded Application to Scrape Working Web Proxies



  



	



---

[![forthebadge made-with-python](http://ForTheBadge.com/images/badges/made-with-python.svg)](https://www.python.org/)

[![Open Source Love png3](https://badges.frapsoft.com/os/v3/open-source.png?v=103)](https://github.com/ellerbrock/open-source-badges/)

[![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://GitHub.com/1UC1F3R616/fastProxy)

## Functionalites

- [x] More than 300 Live Proxy Fetch

- [x] CLI Support

- [x] Selection of Proxy based on Speed

- [x] Proxy Export

- [ ] Country Filteration

### Detailed Documentation

- [Documentation](https://github.com/1UC1F3R616/fastProxy/blob/master/DOCUMENTATION.md)

## Installation

### pip install

```bash

pip install fastProxy==1.0.0

```

### git clone

```text

git clone https://github.com/1UC1F3R616/fastProxy.git

cd fastProxy/

pip install -r requirements.txt -U

```

## Run using CLI

#### Default run

- Threads: 100

- Request Timeout: 4sec

```bash

# Basic usage

python cli.py

# With options

python cli.py --c=10 --t=5 --g --a

```

#### Aletered Parameters

| Flag        | Usage           | Purpose  |  Default  |  Usage  |

| ------------- |:-------------:|:-----:|:-----:|:-----:|

| c     | Thread Count | Increase Testing Speed |   100 | `--c=16`  |

| t      | Request Timeout in sec    |   Give Faster Proxy when set to lower Values | 4 | `--t=20`  |

| g | Generate CSV      |  Generate CSV of Working proxy only with user flags| False | `--g` |

| a | All Scraped Proxy     |  Generate CSV of All Scrapped Proxies with more Detail  | False | `--a` |

## Run by import

- Set Flags or Default Values are Taken

| Flag        | Usage           | Purpose  |  Default  | Usage|

| ------------- |:-------------:|:-----:|:-----:|:-----:|

| c     | Thread Count | Increase Testing Speed |   100 | `c=256`|

| t      | Request Timeout in sec    |   Give Faster Proxy when set to lower Values| 4 | `t=2` |

| g | Generate CSV      |  Generate CSV of Working proxy only with user flags| False | `g=True`|

| a | All Scraped Proxy     |  Generate CSV of All Scrapped Proxies with more Detail  | False | `a=True`|

```py

from fastProxy import fetch_proxies

# Basic usage

proxies = fetch_proxies()

print(proxies)

# With options

proxies = fetch_proxies(c=10, t=5, g=True, a=True)

```

#### Sample [CSV File](https://github.com/1UC1F3R616/fastProxy/blob/master/Sample/all_proxies.csv)

#### TODOs

- [ ] Tag slow tests

- [ ] Fix failing tests

- [ ] Add support for `https://proxyscrape.com/free-proxy-list` using ` https://api.proxyscrape.com/v4/free-proxy-list/get?request=display_proxies&proxy_format=protocolipport&format=json`

- [ ] Remove redundant code and files

- [ ] Refactor linux only code with proper handling

[![LinkedIn](https://img.shields.io/static/v1.svg?label=Connect&message=@Kush&color=grey&logo=linkedin&labelColor=blue&style=social)](https://www.linkedin.com/in/kush-choudhary-567b38169?lipi=urn%3Ali%3Apage%3Ad_flagship3_profile_view_base_contact_details%3BDYkgbUGhTniMSRqOUkdN3A%3D%3D)