https://github.com/x-cold/task-queue-sample
Node.js 生产者/消费者 模式简单实践 - 多进程爬虫任务队列
https://github.com/x-cold/task-queue-sample
Last synced: about 1 month ago
JSON representation
Node.js 生产者/消费者 模式简单实践 - 多进程爬虫任务队列
- Host: GitHub
- URL: https://github.com/x-cold/task-queue-sample
- Owner: x-cold
- Created: 2017-02-20T15:28:02.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2017-02-20T15:35:20.000Z (over 8 years ago)
- Last Synced: 2024-11-11T09:43:55.984Z (7 months ago)
- Language: JavaScript
- Size: 7.81 KB
- Stars: 13
- Watchers: 3
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-github-star - task-queue-sample - 多进程爬虫任务队列 | x-cold | 13 | (JavaScript)
README
### 学习目的
+ 了解和使用生产者/消费者模式
+ 了解任务调度
+ 掌握基本的分布式架构及通讯
+ 掌握Node.js多进程工作机制
### 概念须知
参考文章：http://blog.csdn.net/kaiwii/article/details/6758942
### 生产者/消费者 爬虫
+ 启动缓存队列
```bash
> cd cache
> npm install
> npm run koa
```
+ 启动生产者进程
```bash
> npm install
> node ./producer/cluster.js [进程数量]
```
+ 启动消费者进程
```bash
> npm install
> node ./customer/cluster.js [进程数量]
```