https://github.com/haxzie-xx/crode.js-node-web-crawler

Node.js Crawler built for open FTP sites for movie link collection.
https://github.com/haxzie-xx/crode.js-node-web-crawler

crawler nodejs

Last synced: 12 days ago
JSON representation

Node.js Crawler built for open FTP sites for movie link collection.

• Host: GitHub
• URL: https://github.com/haxzie-xx/crode.js-node-web-crawler
• Owner: haxzie-xx
• License: gpl-3.0
• Created: 2017-06-14T06:34:36.000Z (over 7 years ago)
• Default Branch: master
• Last Pushed: 2017-07-23T18:22:32.000Z (over 7 years ago)
• Last Synced: 2024-11-01T08:42:23.245Z (2 months ago)
• Topics: crawler, nodejs
• Language: JavaScript
• Homepage:
• Size: 915 KB
• Stars: 4
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE
  • Code of conduct: CODE_OF_CONDUCT.md

Awesome Lists containing this project

README

        
# crode.js

Visit [haxzie.me](http://haxzie.me) for demo.

Node.js web crawler scrapping file downloadable link from open ftp servers.

crode.js is an express app.crawling is made possible using [npm - Crawler](https://www.npmjs.com/package/crawler).

app scrapes the html file using npm modules:

-[request](https://www.npmjs.com/package/request)

-[cheerio](https://www.npmjs.com/package/cheerio)

### Storage

crode.js uses Mongo db by default currenty uses only two db schemas for storing download links found and visited.