https://github.com/barretlee/spider

A Spider Man in multiple processes
https://github.com/barretlee/spider

cluster multiple-processes spider

Last synced: 7 months ago
JSON representation

A Spider Man in multiple processes

• Host: GitHub
• URL: https://github.com/barretlee/spider
• Owner: barretlee
• Created: 2016-11-05T17:44:10.000Z (over 9 years ago)
• Default Branch: master
• Last Pushed: 2017-02-04T00:50:31.000Z (about 9 years ago)
• Last Synced: 2025-03-22T22:12:17.323Z (11 months ago)
• Topics: cluster, multiple-processes, spider
• Language: JavaScript
• Homepage:
• Size: 7.81 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
Spider

===

A Web Spider. Grabbing article, images, audio and video in multiple processes.

### Usage

- rename `_config.js` to `config.js`

- set config file

```js

var OPEN_ARTICLE = true;

var OPEN_IMG = true;

var OPEN_VIDEO = true;

```

- start

```bash

node index.js

```

### Todo

- Grab audio and video.

- Complete test case.

### License

MIT.