https://github.com/arstgit/ya-proxy-pool

Yet another proxy pool implementation.
https://github.com/arstgit/ya-proxy-pool

Last synced: 4 months ago
JSON representation

Yet another proxy pool implementation.

• Host: GitHub
• URL: https://github.com/arstgit/ya-proxy-pool
• Owner: arstgit
• License: mit
• Created: 2020-01-18T13:06:28.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2022-12-10T15:31:08.000Z (over 2 years ago)
• Last Synced: 2025-02-28T23:17:12.913Z (4 months ago)
• Language: JavaScript
• Size: 29.3 KB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# ya-proxy-pool

[![Build Status](https://travis-ci.org/derekchuank/ya-proxy-pool.svg?branch=master)](https://travis-ci.org/derekchuank/ya-proxy-pool)

[![npm version](https://badge.fury.io/js/ya-proxy-pool.svg)](http://badge.fury.io/js/ya-proxy-pool)

## Prerequisites

  redis or docker and docker-compose, surely we prefer docker and docker-compose.

```

  apt-get install redis

```

or

```

  apt install docker.io docker-compose

```

## Configfile

  The config file is located in `config/default.json`, you can create another file to rewrite the default config, Seeing `ya-config-loader(https://github.com/derekchuank/ya-config-loader)`.

  - `NAME`. Whatever you like.

  - `REDIS`. Redis config, passed to redis driver package. 

  - `TEST_TARGET`. The URL used to test a proxy usable or not.

  - `SCOUR_INTERVAL`. milliseconds. The testing proxies action time interval.

  - `SCOUR_BATCH`. seconds. Number of proxies of a testing proxies acion.

  - `SPIDER_INTERVAL`. milliseconds. The time interval between crawling acions.

  - `IP_IDLE_INTERVAL`. milliseconds. A proxy can idle some time before tested again.

  - `IP_TIMEOUT`. milliseconds. Http request timeout.

## Usage

  if you have docker and docker-compose installed.

```

  npm run up

```

  or, start a redis-server first, and config your redis related configration, then:

```

  npm start

```

## Data storage

  With docker and docker-compose, your data is writen in directoy `data`. All ips and ports information are stored in a `sorted set` with a key named `${config.NAME}:ip`. In the sorted set, the score means last tested time.

## About proxy resource

  All proxy resource addresses and related retriving methods are written in `src/core.js`. Feel free to modify the file.