{"id":18501513,"url":"https://github.com/gagan1510/greendeck-proxygrabber","last_synced_at":"2025-04-09T01:32:20.806Z","repository":{"id":54838314,"uuid":"201640733","full_name":"gagan1510/greendeck-proxygrabber","owner":"gagan1510","description":"A python library for scraping/checking/fetching/storing proxies. 🎭","archived":false,"fork":false,"pushed_at":"2021-01-26T06:09:04.000Z","size":81,"stargazers_count":20,"open_issues_count":0,"forks_count":5,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-23T20:36:58.690Z","etag":null,"topics":["greendeck","http-proxy","https-proxies","mongodb","proxies","proxies-generator","proxies-http","proxies-scraper","proxy","proxy-server","proxy-service","proxychecker","proxyservice"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gagan1510.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-08-10T14:26:21.000Z","updated_at":"2024-03-18T07:41:12.000Z","dependencies_parsed_at":"2022-08-14T04:20:37.068Z","dependency_job_id":null,"html_url":"https://github.com/gagan1510/greendeck-proxygrabber","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gagan1510%2Fgreendeck-proxygrabber","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gagan1510%2Fgreendeck-proxygrabber/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gagan1510%2Fgreendeck-proxygrabber/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gagan1510%2Fgreendeck-proxygrabber/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gagan1510","download_url":"https://codeload.github.com/gagan1510/greendeck-proxygrabber/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247956973,"owners_count":21024628,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["greendeck","http-proxy","https-proxies","mongodb","proxies","proxies-generator","proxies-http","proxies-scraper","proxy","proxy-server","proxy-service","proxychecker","proxyservice"],"created_at":"2024-11-06T13:54:12.110Z","updated_at":"2025-04-09T01:32:19.853Z","avatar_url":"https://github.com/gagan1510.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"greendeck-proxygrabber 🎭\n---\n![Gd Logo](https://www.greendeck.co/images/logo/logo_full.png \"Greenddeck\")\n\n*This package is developed by [Greendeck](https://www.greendeck.co/)*\n### Install from pip\nhttps://pypi.org/project/greendeck-proxygrabber/\n\n```pip install greendeck-proxygrabber```\n\n---\n**WHATS NEW?**\n\nAdded proxy grabbing support of 4 new regions to proxy service, proxy grabber and proxy scraper.\n\n---\n\n### 👉 What is proxy service?\n\nProxy service is a service that keeps and updates a Mongo Database with latest up and running proxies.\n\n### 👉 How to use?\n\n##### import the service class\n\n```python\nfrom greendeck_proxygrabber import ProxyService\nservice = ProxyService(MONGO_URI = 'mongodb://127.0.0.1:27017',\n                       update_time = 300,\n                       pool_limit = 1000,\n                       update_count = 200,\n                       database_name = 'proxy_pool',\n                       collection_name_http = 'http',\n                       collection_name_https = 'https',\n                       country_code = 'ALL'\n                       )\n```\n\nThis creates a service object.\n\n##### Args\n\n* update_time = Time after which proxies will be updated (in seconds)\n* pool_limit = Limit after which insertion will change to updating\n* update_count = Number of proxies to request grabber at a time\n* database_name = Mongo Database name to store proxies in\n* collection_name_http = Collection name to store http proxies in\n* collection_name_https = Collection name to store https proxies in\n* country_code = ISO code of one of regions supported\n\nList of supported regions is:\n* Combined Regions: ALL\n* United States: US\n* Germany: DE\n* Great Britain: GB\n* France: FR\n* Czech Republic: CZ\n* Netherlands: NL\n* India: IN\n\n#### Starting the service\n\n```python\nservice.start()\n```\n\nStarting service gives the following output:\n\n```Starting proxy service with the following configuration\nMONGO_URI: mongodb://127.0.0.1:27017\nDatabase: proxy_pool\nCollection names: http, https\nPress Ctrl+C once to stop...\nRunning Proxy Service...\n```\n\nThis will run forever and will push/update proxies in mongodb after every {```update_time```} seconds.\n\n### 👉 What is proxy to mongo?\n\nProxy to mongo is a functionality that lets you grab a set of valid proxies from the Internet and store it to the desired MongoDB database. You can schedule this to update or insert a given set of proxies to your database of pool, i.e. put it on airflow or any task scheduler.\n\n### 👉 How to use?\n\n##### import the ProxyToMongo class\n\n```python\nfrom greendeck_proxygrabber import ProxyService\nservice = ProxyToMongo( MONGO_URI = MONGO_URI,\n                        pool_limit = 1000,\n                        length_proxy = 200,\n                        database_name='proxy_pool',\n                        collection_name_http='http',\n                        collection_name_https='https',\n                        country_code='DE'\n                        )\n```\n\nThis creates a service object.\n\n##### Args\n\n* pool_limit = Total number of proxies to keep in mongo/pass None if you don't want to update\n* length_proxy = Number of proxies to fetch at once\n* database_name = Mongo Database name to store proxies in\n* collection_name_http = Collection name to store http proxies in\n* collection_name_https = Collection name to store https proxies in\n* country_code = ISO code of one of regions supported\n\nList of supported regions is:\n* Combined Regions: ALL\n* United States: US\n* Germany: DE\n* Great Britain: GB\n* France: FR\n* Czech Republic: CZ\n* Netherlands: NL\n* India: IN\n\n#### Calling the ProxyToMongo grabber\n\n```python\nservice.get_quick_proxy()\n```\n\nStarting Grabber gives the following output:\n\n```Gathering proxies with the following configuration:\nMONGO_URI: mongodb://127.0.0.1:27017\nDatabase: proxy_pool\nCollection names: http, https\nPress Ctrl+C once to stop...\nRunning Proxy Grabber...\n```\n\nThis will run forever and will push/update proxies in mongodb after every {```update_time```} seconds.\n\n### 👉 How to use Proxy Grabber Class?\n\n##### import ```ProxyGrabber``` class\n```python\nfrom greendeck_proxygrabber import ProxyGrabber\n```\n\n##### initialize ```ProxyGrabber``` object\n```python\ngrabber = ProxyGrabber(len_proxy_list, country_code, timeout)\n```\nHere default values of some arguments are,\n```\nlen_proxy_list = 10\ncountry_code = 'ALL'\ntimeout = 2\n```\nCurrently the program only supports proxies of combined regions\n\n##### Getting checked, running proxies\nThe grab_proxy ```grab_proxy()``` function helps to fetch the proxies.\n```python\ngrabber.grab_proxy()\n```\nThis returns a dictionary of the following structure:\n```python\n{\n    'https': [\u003c list of https proxies \u003e],\n    'http': [\u003c list of http proxies \u003e],\n    'region': 'ALL' # default for now\n}\n```\n##### Getting an unchecked list of proxies\nThe grab_proxy ```proxy_scraper()``` method of ```ScrapeProxy``` helps to fetch the proxies.\nThis returns a list of 200 proxies of both type http and https.\n```python\nfrom greendeck_proxygrabber import ScrapeProxy\nproxies_http, proxies_https = ScrapeProxy.proxy_scraper()\n```\nThis returns list of proxies of type http proxies followed by https proxies.\n```\nhttp_proxies = [\u003c list of http proxies \u003e]\nhttps_proxies = [\u003c list of https proxies \u003e]\n```\n##### Filtering invalid proxies from a list of proxies\nThe ```proxy_checker_https``` and ```proxy_checker_http``` methods from ```ProxyChecker``` class helps to validate the proxies.\n\nGiven a list of proxies, it checks each of them to be valid or not, and returns a list of valid proxies from the proxies feeded to it.\n\n```python\nfrom greendeck_proxygrabber import ProxyChecker\nvalid_proxies_http = ProxyChecker.proxy_checker_http(proxy_list = proxy_list_http, timeout = 2)\nvalid_proxies_https = ProxyChecker.proxy_checker_https(proxy_list = proxy_list_https, timeout = 2)\n```\n\n---\n👉 How to build your own pip package\n\n* open an account here https://pypi.org/\n\nIn the parent directory\n* ```python setup.py sdist bdist_wheel```\n* ```twine upload dist/*```\n\nreferences\n* https://medium.com/small-things-about-python/lets-talk-about-python-packaging-6d84b81f1bb5\n\n\nMADE WITH 🐍 BY Gagan\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgagan1510%2Fgreendeck-proxygrabber","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgagan1510%2Fgreendeck-proxygrabber","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgagan1510%2Fgreendeck-proxygrabber/lists"}