https://github.com/skykery/scraping_stuff
https://github.com/skykery/scraping_stuff
Last synced: 5 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/skykery/scraping_stuff
- Owner: skykery
- Created: 2022-04-15T13:13:17.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2022-04-15T13:17:38.000Z (about 4 years ago)
- Last Synced: 2024-12-30T00:34:53.046Z (over 1 year ago)
- Language: Python
- Size: 1000 Bytes
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Examples for web scraping and crawling
## Useful resources
### Basic Knowledge
- [CSS Selectors](https://www.w3schools.com/cssref/css_selectors.asp)
- [HTML DOM](https://www.w3schools.com/whatis/whatis_htmldom.asp)
### Core
- [lxml](https://lxml.de/)
- [cssselect](https://cssselect.readthedocs.io/en/latest/) (required for lxml if css selectors are used)
- [requests](https://docs.python-requests.org/en/latest/)
### Extra
- [splash](https://splash.readthedocs.io/en/stable/)
- [ratelimiter](https://pypi.org/project/ratelimiter/)
- [requests-cache](https://requests-cache.readthedocs.io/en/stable/)
- [tablib](https://tablib.readthedocs.io/en/stable/)
### Some day
These are used for more complicated crawlers.
- [celery](https://github.com/celery/celery)
- [rabbitmq](https://www.rabbitmq.com/)
- [scrapy](https://scrapy.org/)