https://github.com/zhs007/crawler-js

a simple crawler
https://github.com/zhs007/crawler-js

Last synced: 5 days ago
JSON representation

a simple crawler

• Host: GitHub
• URL: https://github.com/zhs007/crawler-js
• Owner: zhs007
• License: mit
• Created: 2017-05-05T02:23:49.000Z (about 9 years ago)
• Default Branch: master
• Last Pushed: 2017-12-04T03:33:14.000Z (over 8 years ago)
• Last Synced: 2025-02-22T21:14:13.780Z (over 1 year ago)
• Language: JavaScript
• Size: 109 KB
• Stars: 1
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# crawler-js

一个nodejs爬虫，以下特点：

1. 组件式，可配置出各种类型的爬虫

2. 使用async/await

3. 自动的编码处理

4. 支持headless chrome

5. 可配置的持久化模块

6. 爬虫管理器