https://github.com/akagi201/spy

A lightweight distributed web crawler
https://github.com/akagi201/spy

crawler distributed lightweight nsq

Last synced: over 1 year ago
JSON representation

A lightweight distributed web crawler

• Host: GitHub
• URL: https://github.com/akagi201/spy
• Owner: Akagi201
• License: mit
• Created: 2017-08-22T04:54:27.000Z (almost 9 years ago)
• Default Branch: master
• Last Pushed: 2017-08-22T06:34:10.000Z (almost 9 years ago)
• Last Synced: 2025-01-08T23:26:07.623Z (over 1 year ago)
• Topics: crawler, distributed, lightweight, nsq
• Language: Go
• Size: 16.6 KB
• Stars: 2
• Watchers: 1
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# spy

![spy](assets/spy.jpg)

A lightweight distributed web crawler.

## Features

- [ ] HTML parsing and extracting with [goquery](https://github.com/PuerkitoBio/goquery).

- [ ] Support Go1.7 net/context.

- [ ] Distributed architecture wit NSQ.