https://github.com/william-fernandes252/astel

An asyncronous web crawling library for Python.
https://github.com/william-fernandes252/astel

async async-task asyncio httpx python robots-txt web-crawler web-crawling

Last synced: 12 months ago
JSON representation

An asyncronous web crawling library for Python.

• Host: GitHub
• URL: https://github.com/william-fernandes252/astel
• Owner: William-Fernandes252
• License: mit
• Created: 2023-05-25T00:58:11.000Z (over 2 years ago)
• Default Branch: main
• Last Pushed: 2024-05-13T19:05:25.000Z (almost 2 years ago)
• Last Synced: 2025-02-04T10:02:45.377Z (about 1 year ago)
• Topics: async, async-task, asyncio, httpx, python, robots-txt, web-crawler, web-crawling
• Language: Python
• Homepage: https://william-fernandes252.github.io/astel/
• Size: 1.02 MB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE

Awesome Lists containing this project

README

          


  



# Astel

[![pypi](https://img.shields.io/pypi/v/astel.svg)](https://pypi.org/project/astel/)

[![python](https://img.shields.io/pypi/pyversions/astel.svg)](https://pypi.org/project/astel/)

[![Build Status](https://github.com/William-Fernandes252/astel/actions/workflows/dev.yml/badge.svg)](https://github.com/William-Fernandes252/astel/actions/workflows/dev.yml)

[![codecov](https://codecov.io/gh/William-Fernandes252/astel/graph/badge.svg?token=D8S6AF7A3Q)](https://codecov.io/gh/William-Fernandes252/astel)

[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)

A simple, fast and reliable asyncronous web crawler for Python.

* Documentation: 

* GitHub: 

* PyPI: 

* Free software: MIT

## Features

The main goal of `astel` is to offer a simpler, efficient and performant solution to programmatically look for

links  in webpages: no need to extend any class (**composition** over inheritance), no configuration and as few dependencies as possible.

This package relies on [HTTPX](https://www.python-httpx.org/) to send all requests in asynchronous operations, thus maximizing the number of pages processed during each execution.

## Credits

This package was created with [Cookiecutter](https://github.com/audreyr/cookiecutter) and the [waynerv/cookiecutter-pypackage](https://github.com/waynerv/cookiecutter-pypackage) project template.