https://github.com/montenegrodr/letmecrawl

Curated free proxies
https://github.com/montenegrodr/letmecrawl

crawler proxy proxy-server proxypool scraper

Last synced: 5 months ago
JSON representation

Curated free proxies

• Host: GitHub
• URL: https://github.com/montenegrodr/letmecrawl
• Owner: montenegrodr
• License: mit
• Created: 2017-12-30T20:05:18.000Z (over 8 years ago)
• Default Branch: master
• Last Pushed: 2023-05-01T20:19:52.000Z (about 3 years ago)
• Last Synced: 2025-08-20T14:35:25.174Z (10 months ago)
• Topics: crawler, proxy, proxy-server, proxypool, scraper
• Language: Python
• Size: 25.4 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 1
• Metadata Files:
  • Readme: README.rst
  • License: LICENSE.txt

Awesome Lists containing this project

README

          
Let me crawl

=======================

Curate proxies from various sources in terms of latency. Supported on python 2.7.* and >3.4.

.. code-block:: bash

    >> from letmecrawl import letmecrawl

    >> for proxy in letmecrawl():

    >>     print(proxy)

    88.99.11.10:6666

Installation

-----------------

.. code-block:: bash

    pip install letmecrawl

Docker

-----------------

Use docker image as a http service:

.. code-block:: bash

    docker pull montenegrodr/letmecrawl:latest

Running:

.. code-block:: bash

    docker run -p 8000:8000 montenegrodr/letmecrawl:latest

Request proxy from top of the queue:

.. code-block:: bash

    $ curl localhost:8000

    {"proxy":"138.68.178.106:80"}

    $ curl localhost:8000

    {"proxy":"184.59.102.95:80"}