https://github.com/buckyroberts/Spider

Python website crawler.
https://github.com/buckyroberts/Spider

Last synced: 15 days ago
JSON representation

Python website crawler.

• Host: GitHub
• URL: https://github.com/buckyroberts/Spider
• Owner: buckyroberts
• Created: 2016-02-24T21:44:33.000Z (over 8 years ago)
• Default Branch: master
• Last Pushed: 2023-01-02T10:54:11.000Z (almost 2 years ago)
• Last Synced: 2024-08-01T15:14:22.393Z (3 months ago)
• Language: Python
• Homepage: https://thenewboston.com/
• Size: 45.9 KB
• Stars: 965
• Watchers: 113
• Forks: 674
• Open Issues: 83
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• awesome - Spider - Python website crawler. (Python)

README

        
![](http://i.imgur.com/wYi2CkD.png)

# Overview

This is an open source, multi-threaded website crawler written in Python. There is still a lot of work to do, so feel free to help out with development.

***

Note: This is part of an open source search engine. The purpose of this tool is to gather links **only**. The analytics, data harvesting, and search algorithms are being created as separate programs. 

### Links

- [Support thenewboston](https://www.patreon.com/thenewboston)

- [thenewboston.com](https://thenewboston.com/)

- [Facebook](https://www.facebook.com/TheNewBoston-464114846956315/)

- [Twitter](https://twitter.com/bucky_roberts)

- [Google+](https://plus.google.com/+BuckyRoberts)

- [reddit](https://www.reddit.com/r/thenewboston/)