https://github.com/buckyroberts/Spider

Python website crawler.
https://github.com/buckyroberts/Spider

Last synced: 21 days ago
JSON representation

Python website crawler.

• Host: GitHub
• URL: https://github.com/buckyroberts/Spider
• Owner: buckyroberts
• Created: 2016-02-24T21:44:33.000Z (over 8 years ago)
• Default Branch: master
• Last Pushed: 2023-01-02T10:54:11.000Z (over 1 year ago)
• Last Synced: 2024-02-17T22:36:14.297Z (4 months ago)
• Language: Python
• Homepage: https://thenewboston.com/
• Size: 45.9 KB
• Stars: 956
• Watchers: 114
• Forks: 732
• Open Issues: 84
• Metadata Files:
  • Readme: README.md

Lists

• awesome-stars - buckyroberts/Spider - Python website crawler. (Python)
• awesome - Spider - Python website crawler. (Python)
• awesome-stars - buckyroberts/Spider - Python website crawler. (Python)
• awesome-stars - buckyroberts/Spider - Python website crawler. (Python)
• my-awesome-stars - buckyroberts/Spider - Python website crawler. (Python)

README

        
![](http://i.imgur.com/wYi2CkD.png)

# Overview

This is an open source, multi-threaded website crawler written in Python. There is still a lot of work to do, so feel free to help out with development.

***

Note: This is part of an open source search engine. The purpose of this tool is to gather links **only**. The analytics, data harvesting, and search algorithms are being created as separate programs. 

### Links

- [Support thenewboston](https://www.patreon.com/thenewboston)

- [thenewboston.com](https://thenewboston.com/)

- [Facebook](https://www.facebook.com/TheNewBoston-464114846956315/)

- [Twitter](https://twitter.com/bucky_roberts)

- [Google+](https://plus.google.com/+BuckyRoberts)

- [reddit](https://www.reddit.com/r/thenewboston/)