https://github.com/panagiks/asset

ASynchronous Spidering Essential Tool (ASSET).
https://github.com/panagiks/asset

async asyncio crawler graph reporting spider

Last synced: 5 days ago
JSON representation

ASynchronous Spidering Essential Tool (ASSET).

• Host: GitHub
• URL: https://github.com/panagiks/asset
• Owner: panagiks
• License: mit
• Created: 2017-07-01T08:43:54.000Z (over 7 years ago)
• Default Branch: master
• Last Pushed: 2017-11-29T11:09:51.000Z (about 7 years ago)
• Last Synced: 2024-12-06T20:56:55.034Z (2 months ago)
• Topics: async, asyncio, crawler, graph, reporting, spider
• Language: Python
• Homepage:
• Size: 17.6 KB
• Stars: 1
• Watchers: 3
• Forks: 2
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# ASSET

ASynchronous Spidering Essential Tool (ASSET).

# Status

ASSET is currently in a PoC (Proof of Concept) stage. Feel free to toy around with it

and explore its potential, just take note that none of the API is documented or even

finalized.

# Todo

- [x] Add CLI interactions

- [ ] Add database relational model to store the crawled Pages

- [ ] Add RESTful API to provide data from said database

- [ ] Create better looking (and maybe interactive) graphs

- [ ] Don't recheck the same URL with different protocol