https://github.com/hominee/dyer

Dyer is designed for reliable, flexible and fast web crawling, providing some high-level, comprehensive features without compromising speed.
https://github.com/hominee/dyer

crawler rust rust-programming-language spider web-crawler web-framework web-scraping

Last synced: 3 months ago
JSON representation

Dyer is designed for reliable, flexible and fast web crawling, providing some high-level, comprehensive features without compromising speed.

• Host: GitHub
• URL: https://github.com/hominee/dyer
• Owner: hominee
• Created: 2021-02-01T04:16:21.000Z (almost 4 years ago)
• Default Branch: master
• Last Pushed: 2022-11-23T01:14:46.000Z (almost 2 years ago)
• Last Synced: 2024-03-15T22:04:59.769Z (8 months ago)
• Topics: crawler, rust, rust-programming-language, spider, web-crawler, web-framework, web-scraping
• Language: Rust
• Homepage: https://hominee.github.io/dyer/
• Size: 75 MB
• Stars: 134
• Watchers: 10
• Forks: 14
• Open Issues: 1
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• awesome-rust - hominee/dyer - dyer is designed for reliable, flexible and fast Request-Response based service, including data processing, web-crawling and so on, providing some friendly, flexible, comprehensive features without compromising speed. (Libraries / Web programming)
• awesome-rust-cn - hominee/dyer - (库 Libraries / 网络编程 Web programming)

README

        
# [dyer](https://github.com/hominee/dyer)

[![API Document](https://img.shields.io/docsrs/dyer/latest)](https://docs.rs/dyer)

[![crates.io](https://img.shields.io/crates/v/dyer.svg)](https://crates.io/crates/dyer)

[![Cookbook](https://img.shields.io/static/v1?label=cookbook&message=dyer&color=green)](https://hominee.github.io/dyer/)

## Overview

[dyer] is designed for reliable, flexible and fast Request-Response based service, including data processing, web-crawling and so on, providing some friendly,  interoperable, comprehensive  features without compromising speed.

## Features

* asynchronous, concurrent streaming and I/O, make the best of thread pool, network, and system

resource.

* Event-driven, once you set the initials and recursive generator, `dyer` will handle

the rest of it interoperably.

* User-friendly and flexible, `dyer` offers high-level, flexible, easy to use wrappers and APIs what does a lot for you.    

## Feature Flag

To reduce code redundancy and speed up compilation, dyer use feature flag to mark the necessary modules/functions, Currently here are some supported Features:

- `xpath-stable`: stably parse the response html with third-party C library `libxml2`   

- `xpath-alpha`: rust-native parse the html response, **NOTE** that it is experimental and unstale, remains to be improved  

- `compression`: Enable HTTP Compression: `br`, `deflate`, `gzip`

- `proxy`: Enable use proxies

- `full`: Enable all features

## Guide

**Get started** by installing [dyer-cli] and looking over the [examples] and [quick start].

Crates: [Link 🔗](https://crates.io/crates/dyer/)   

Documentation: [Link 🔗](https://docs.rs/dyer/latest/dyer)   

The [Cookbook](https://hominee.github.io/dyer/) gives a detailed view of dyer.

[dyer]: https://docs.rs/dyer

[examples]: https://github.com/hominee/dyer/tree/master/examples/

[quick start]: https://github.com/hominee/dyer/tree/master/quick-start.md/

[dyer-cli]: https://github.com/hominee/dyer-cli