https://github.com/nextlevelshit/adonis-crawler

A free web crawler on top of the incredibile AdonisJS Framework
https://github.com/nextlevelshit/adonis-crawler

adonisjs crawler javascript nodejs regex spider websocket

Last synced: 21 days ago
JSON representation

A free web crawler on top of the incredibile AdonisJS Framework

• Host: GitHub
• URL: https://github.com/nextlevelshit/adonis-crawler
• Owner: nextlevelshit
• Created: 2017-03-06T15:12:39.000Z (almost 8 years ago)
• Default Branch: master
• Last Pushed: 2017-03-08T08:49:41.000Z (almost 8 years ago)
• Last Synced: 2024-11-19T19:18:48.575Z (3 months ago)
• Topics: adonisjs, crawler, javascript, nodejs, regex, spider, websocket
• Language: JavaScript
• Size: 40 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md

Awesome Lists containing this project

README

        
# Adonis Web Crawler

A free web crawler on top of the incredibile AdonisJS Framework

## Road map

Things wanting to get done.

- [x] Working draft

- [x] Recursive search

- [x] Pause and continue button

- [ ] Normalize links by using the server response

- [ ] Normalize links by removing trailing hashes

- [ ] Convert links inside `CrawlerController` to models

- [ ] Set own regex for extracting data

- [ ] Write tests

## Author

Michael Czechowski 

_Dependencies and external libraries are listed in the [`package.json`](https://github.com/nextlevelshit/adonis-crawler/blob/master/package.json#L19)_

## License

```

Adonis Crawler - A free web crawler on top of AdonisJs Framework

Copyright (C) 2017 Michael Czechowski

This program is free software: you can redistribute it and/or modify

it under the terms of the GNU General Public License as published by

the Free Software Foundation, either version 3 of the License, or

(at your option) any later version.

This program is distributed in the hope that it will be useful,

but WITHOUT ANY WARRANTY; without even the implied warranty of

MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

GNU General Public License for more details.

You should have received a copy of the GNU General Public License

along with this program.  If not, see .

```