https://github.com/meain/mucri
Quickly fetch a lot of pages/apis using python asyncio
https://github.com/meain/mucri
Last synced: about 1 month ago
JSON representation
Quickly fetch a lot of pages/apis using python asyncio
- Host: GitHub
- URL: https://github.com/meain/mucri
- Owner: meain
- Created: 2018-10-30T11:00:52.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2019-05-02T11:34:24.000Z (almost 6 years ago)
- Last Synced: 2025-03-19T02:11:38.581Z (about 1 month ago)
- Language: Python
- Size: 9.77 KB
- Stars: 6
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Mucri
Quickly fetch a lot of pages/apis using python `asyncio`.
## Installation
> Only python 3.6+
```sh
pip isntall mucri
```
## Usage
`fetch_pages` takes two args:
`links` : list of links to be fetched (example below)
`concurrency`: how many requests to be send at a time (default 20)
```python
from mucri import fetch_pages
# links can be a single string or a dict with specific instructions
links = [
"http://meain.github.io", # Can be just a url
{ "url": "http://somelink" }, # or a dict with url key
{ # for more complex stuff
"url": "http://fakelink",
"action": "get", # get | post
"data": {},
"headers": {},
"resp_type": "text", # text | json | image
}
]
results = fetch_pages(links) # fetches all of them asynchronously
```