https://github.com/joshlk/many_requests

Dead easy interface for executing many HTTP requests asynchronously. Also provides helper functions for executing embarrassingly parallel async coroutines.
https://github.com/joshlk/many_requests

async python requests

Last synced: 15 days ago
JSON representation

Dead easy interface for executing many HTTP requests asynchronously. Also provides helper functions for executing embarrassingly parallel async coroutines.

• Host: GitHub
• URL: https://github.com/joshlk/many_requests
• Owner: joshlk
• License: mit
• Created: 2020-11-11T11:55:24.000Z (about 4 years ago)
• Default Branch: main
• Last Pushed: 2021-03-23T10:54:45.000Z (over 3 years ago)
• Last Synced: 2024-10-15T15:22:54.613Z (29 days ago)
• Topics: async, python, requests
• Language: Python
• Homepage:
• Size: 5.77 MB
• Stars: 387
• Watchers: 9
• Forks: 8
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • Contributing: docs/contributing.html
  • License: LICENSE

Awesome Lists containing this project

README

        
[![PyPI](https://img.shields.io/pypi/v/dataclassframe)](https://pypi.org/project/many_requests/)

![Python](https://img.shields.io/badge/python-3.6%20%7C%203.7%20%7C%203.8%20%7C%203.9-blue)

[![Build Status](https://travis-ci.com/joshlk/many_requests.svg?branch=main)](https://travis-ci.com/joshlk/many_requests)

[![Documentation](https://readthedocs.org/projects/pip/badge/?version=latest&style=flat)](https://joshlk.github.io/many_requests)

# many_requests

Dead easy interface for executing many HTTP requests asynchronously.

It has been tested in the wild with over 10 million requests.

Automatically handles errors and executes retries.

Built on-top of [Trio](https://github.com/python-trio/trio) and [asks](https://github.com/theelous3/asks). Interface heavily inspired by [Requests](https://github.com/psf/requests) and [joblib](https://github.com/joblib/joblib).

Also provides helper functions for executing [embarrassingly parallel](https://en.wikipedia.org/wiki/Embarrassingly_parallel) async coroutines.

To install:

```bash

pip install many-requests

```

## Example Usage

Execute 10 GET requests for example.org:

```python

from many_requests import ManyRequests

responses = ManyRequests(n_workers=5, n_connections=5)(

                method='GET',

                url=['https://example.org' for i in range(10)])

```

Query HackNews API for 10 items and parse JSON output:

```python

responses = ManyRequests(n_workers=5, n_connections=5, json=True)(

                method='GET',

                url=[f'https://hacker-news.firebaseio.com/v0/item/{i}.json?print=pretty' for i in range(10)])

```

To use basic authentication with all requests:

```python

from asks import BasicAuth

username = 'user'

password = 'pw'

responses = ManyRequests(n_workers=5, n_connections=5)(

                method='GET',

                url=['https://example.org' for i in range(10)],

                auth=BasicAuth((username, password)))

```

To execute embarrassingly parallel async coroutines, for example 10 `trio.sleep` calls:

```python

from many_requests import EasyAsync, delayed

import trio

outputs = EasyAsync(n_workers = 4)(delayed(trio.sleep)(i) for i in range(10))

```