https://github.com/schafeld/crawler
Website crawler, Python
https://github.com/schafeld/crawler
python python3
Last synced: about 1 year ago
JSON representation
Website crawler, Python
- Host: GitHub
- URL: https://github.com/schafeld/crawler
- Owner: schafeld
- Created: 2019-04-21T13:06:41.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2019-04-21T14:00:04.000Z (over 7 years ago)
- Last Synced: 2025-06-02T22:34:16.641Z (about 1 year ago)
- Topics: python, python3
- Language: Python
- Size: 3.91 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# crawler
Website crawler, Python
## Installation
Check out from Github
Install dependencies/modules
``` pip install beautifulsoup4 ```
or (if like me you have Python 2.x and Python 3.x)
``` pip3 install bs4 ```
``` pip3 install requests ```
``` pip3 install lxml ```
Execution
```python3 crawler.py```
## Acknowledgment
Thanks for source/inspiration: https://medium.freecodecamp.org/how-to-build-a-url-crawler-to-map-a-website-using-python-6a287be1da11
## Final words
Use this script at your own risk. Crawling websites puts load on them and may be seen as aggressive act, i.e. can get you banned, blocked or worse. As always in life: Don't be rude to people!