Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/JoseVL92/web-requester
HTTP sync / sync library that works with both: requests and aiohttp, exploiting the best of each one
https://github.com/JoseVL92/web-requester
Last synced: about 2 months ago
JSON representation
HTTP sync / sync library that works with both: requests and aiohttp, exploiting the best of each one
- Host: GitHub
- URL: https://github.com/JoseVL92/web-requester
- Owner: JoseVL92
- License: mit
- Created: 2019-09-03T13:15:38.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2022-05-19T19:27:20.000Z (over 2 years ago)
- Last Synced: 2024-07-09T13:30:55.728Z (6 months ago)
- Language: Python
- Size: 40 KB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
- cuban-opensource - http-requests
README
# web-requester
## Easily and efficiently perform concurrent http requests with Python
Make several requests with common options for all, or specific options for each of them.
**aiohttp** library will be used only for default behavior (text retrieving, no callback) and if not explicitly disabled via `{'allow_aio' = False}` in common_options. **requests** will be used otherwise
Options dictionary has shape (for example):
```json
{
"method": "get",
"data": b"Hola Mundo!!!",
"params": {"q": "frase a buscar"},
"json": [{ "si": "no" }],
"proxy_cfg": {"http": "http://miproxy"},
"headers": {"Content-Type": "application/pdf"},
"allow_aio": True,
"aio_client": some_aiohttp_client,
"timeout": 15,
"logger": some_logger,
"callback": some_function
}
```
### Some points to notice:
- **method**: 'get' by default
- **data**: used only in non-get requests
- **params**: used only in 'get' requests
- **proxy**: could have also the "http://user:pass@host:port" format
- **allow_aio**: enables or disables the use of _aiohttp_. If False, it ensures the use of _requests_ always and excludes _aiohttp_. Default: True
- **aio_client**: _aiohttp.ClientSession_. It's the only option that, if present, must be global, not specific for a single url
- **logger**: a _logging.logger_ instance
- **callback**: _function_ that receives a _requests.Response_ as argument
> _If callback exists, the URL will be fetched using 'requests' only for standardization_
**Args:**
* **urloptions_list**: tuple (url, opts), where 'url' is the url of the request, and opts is a dictionary with specific options
* **common_options**: dictionary with common options for every request, if that request does not specify its own options
**Returns**: A list of responses defined by each specific callback or the general callback, or each response text by default
### How to use
```python
from web_requester import request_all
urls = ['https://google.com',
('https://actualidad.rt.com', {'json': {'num': 2}}),
'http://www.cubadebate.cu',
('https://www.filehorse.com/es', {'timeout': 12})]
response_list = request_all(urls, {'method': 'get', 'timeout': '10'})
```