https://github.com/pustovitdmytro/parser
tiny universal Python-based asynchronous sites scrapper
https://github.com/pustovitdmytro/parser
aiohttp boilerplate parser python3 scrapper
Last synced: 11 months ago
JSON representation
tiny universal Python-based asynchronous sites scrapper
- Host: GitHub
- URL: https://github.com/pustovitdmytro/parser
- Owner: pustovitDmytro
- Created: 2017-09-10T15:05:45.000Z (almost 9 years ago)
- Default Branch: master
- Last Pushed: 2021-10-31T23:30:24.000Z (over 4 years ago)
- Last Synced: 2025-01-10T01:53:33.128Z (over 1 year ago)
- Topics: aiohttp, boilerplate, parser, python3, scrapper
- Language: Python
- Size: 154 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# parser
tiny universal Python-based asynchronous sites scrapper
## Table of Contents
1. [Requirements](#requirements)
2. [Install & Run](#install-&-run)
3. [Project Structure](#project-structure)
4. [Tests](#tests)
5. [Related Projects](related-projects)
6. [License](license)
## Requirements
* python `3.0+`
* pip `9.0+`
## Install & Run
To install packages run:
```sh
$ pip install bs4 selenium aiohttp asyncio async_timeout time abc re
$ python script.py
```
## Project Structure
```
.
├── html # folder with files for offline parsing
│ └── google.example.page # page with google request
├── results # folder with parsed files
├── screenshots # folder with selenium screenshots
├── main.py # file with method implementation
└── test.google.py # example of using
```
## Tests
Best way to test web-parser, is to parse some site with data templates. For instance in `test.google.py` you can explore a way of parsing google search pages.
Run it by
```
$ python test.google.py
```
## Related Projects
There are some more boilerplates that I'm working on:
1. [gulp](https://github.com/pustovitDmytro/gulp) - simple template for quick site creation via gulp
2. [web-extension](https://github.com/pustovitDmytro/web-extension) - boilerplate to making web-extensions (firefox and chrome simultaneously)
3. [react](https://github.com/pustovitDmytro/react) - boilerplate for making gracefull react apps
## Examples Of Use
There are some examples of implementation:
* [dmsu-bot](https://github.com/pustovitDmytro/dmsu-bot) - bot which helped me to order biometric passport
* [quotes](https://github.com/pustovitDmytro/quotes) - scraps famous quotes and saves them to scv
## License
MIT