Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/geofmureithi/apalis

Simple, extensible multithreaded background job and message processing library for Rust
https://github.com/geofmureithi/apalis

background-jobs job-scheduler message-queue mysql postgres redis rust sqlite

Last synced: 15 days ago
JSON representation

Simple, extensible multithreaded background job and message processing library for Rust

Awesome Lists containing this project

README

        

apalis




Simple, extensible multithreaded background job and messages processing library for Rust





Crates.io version



Download



docs.rs docs


CI



## Features

- Simple and predictable job handling model.
- Jobs handlers with a macro free API.
- Take full advantage of the [tower] ecosystem of
middleware, services, and utilities.
- Runtime agnostic - Use tokio, smol etc.
- Optional Web interface to help you manage your jobs.

apalis job processing is powered by [`tower::Service`] which means you have access to the [tower] middleware.

apalis has support for:

| Source | Crate | Example |
| ------------ | --------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Cron Jobs | | |
| Redis | | |
| Sqlite | | |
| Postgres | | |
| MySQL | | |
| Amqp | | |
| From Scratch | | |

## Getting Started

To get started, just add to Cargo.toml

```toml
[dependencies]
apalis = { version = "0.5", features = ["redis"] } # Backends available: postgres, sqlite, mysql, amqp
```

## Usage

```rust
use apalis::prelude::*;
use apalis::redis::RedisStorage;
use serde::{Deserialize, Serialize};
use anyhow::Result;

#[derive(Debug, Deserialize, Serialize)]
struct Email {
to: String,
}

impl Job for Email {
const NAME: &'static str = "apalis::Email";
}

/// A function that will be converted into a service.
async fn send_email(job: Email, data: Data) -> Result<(), Error> {
/// execute job
Ok(())
}

#[tokio::main]
async fn main() -> Result<()> {
std::env::set_var("RUST_LOG", "debug");
env_logger::init();
let redis_url = std::env::var("REDIS_URL").expect("Missing env variable REDIS_URL");
let storage = RedisStorage::new(redis).await?;
Monitor::new()
.register_with_count(2, {
WorkerBuilder::new(format!("email-worker"))
.data(0usize)
.with_storage(storage)
.build_fn(send_email)
})
.run()
.await
}

```

Then

```rust
//This can be in another part of the program or another application eg a http server
async fn produce_route_jobs(storage: &RedisStorage) -> Result<()> {
let mut storage = storage.clone();
storage
.push(Email {
to: "[email protected]".to_string(),
})
.await?;
}

```

## Feature flags

- _tracing_ (enabled by default) — Support Tracing 👀
- _redis_ — Include redis storage
- _postgres_ — Include Postgres storage
- _sqlite_ — Include SQlite storage
- _mysql_ — Include MySql storage
- _cron_ — Include cron job processing
- _sentry_ — Support for Sentry exception and performance monitoring
- _prometheus_ — Support Prometheus metrics
- _retry_ — Support direct retrying jobs
- _timeout_ — Support timeouts on jobs
- _limit_ — 💪 Limit the amount of jobs
- _filter_ — Support filtering jobs based on a predicate

## Storage Comparison

Since we provide a few storage solutions, here is a table comparing them:

| Feature | Redis | Sqlite | Postgres | Sled | Mysql | Mongo | Cron |
| :-------------- | :---: | :----: | :------: | :--: | :---: | :---: | :--: |
| Scheduled jobs | ✓ | ✓ | ✓ | x | ✓ | x | ✓ |
| Retry jobs | ✓ | ✓ | ✓ | x | ✓ | x | ✓ |
| Persistence | ✓ | ✓ | ✓ | x | ✓ | x | BYO |
| Rerun Dead jobs | ✓ | ✓ | ✓ | x | ✓ | x | x |

## How apalis works

Here is a basic example of how the core parts integrate

```mermaid
sequenceDiagram
participant App
participant Worker
participant Backend

App->>+Backend: Add job to queue
Backend-->>+Worker: Job data
Worker->>+Backend: Update job status to 'Running'
Worker->>+App: Started job
loop job execution
Worker-->>-App: Report job progress
end
Worker->>+Backend: Update job status to 'completed'
```

## External examples

- [Shuttle](https://github.com/shuttle-hq/shuttle-examples/tree/main/shuttle-cron): Using apalis-cron with [shuttle.rs](https://shuttle.rs)
- [Actix-Web](https://github.com/actix/examples/tree/master/background-jobs): Using apalis-redis with actix-web

## Projects using apalis

- [Ryot](https://github.com/IgnisDa/ryot): A self hosted platform for tracking various facets of your life - media, fitness etc.
- [Summarizer](https://github.com/akhildevelops/summarizer): Podcast summarizer
- [Universal Inbox](https://github.com/universal-inbox/universal-inbox): Universal Inbox is a solution that centralizes all your notifications and tasks in one place to create a unique inbox.

## Resources

- [Background job processing with rust using actix and redis](https://mureithi.me/blog/background-job-processing-with-rust-actix-redis)

### Web UI

If you are running [apalis Board](https://github.com/geofmureithi/apalis-board), you can easily manage your jobs. See a working [rest API example here](https://github.com/geofmureithi/apalis/tree/master/examples/rest-api)

## Thanks to

- [tower] - Tower is a library of modular and reusable components for building robust networking clients and servers.
- [redis-rs](https://github.com/mitsuhiko/redis-rs) - Redis library for rust
- [sqlx](https://github.com/launchbadge/sqlx) - The Rust SQL Toolkit

## Roadmap

v 0.5

- [x] Refactor the crates structure
- [x] Mocking utilities
- [ ] Support for SurrealDB and Mongo
- [ ] Lock free for Postgres
- [x] Add more utility layers
- [x] Use extractors in job fn structure
- [x] Polish up documentation
- [ ] Improve and standardize apalis Board
- [ ] Benchmarks

v 0.4

- [x] Move from actor based to layer based processing
- [x] Graceful Shutdown
- [x] Allow other types of executors apart from Tokio
- [x] Mock/Test Worker
- [x] Improve monitoring
- [x] Add job progress via layer
- [x] Add more sources

v 0.3

- [x] Standardize API (Storage, Worker, Data, Middleware, Context )
- [x] Introduce SQL
- [x] Implement layers for Sentry and Tracing.
- [x] Improve documentation
- [x] Organized modules and features.
- [x] Basic Web API Interface
- [x] Sql Examples
- [x] Sqlx migrations

v 0.2

- [x] Redis Example
- [x] Actix Web Example

## Contributing

Please read [CONTRIBUTING.md](CONTRIBUTING.md) for details on our code of conduct, and the process for submitting pull requests to us.

## Versioning

We use [SemVer](http://semver.org/) for versioning. For the versions available, see the [tags on this repository](https://github.com/geofmureithi/apalis/tags).

## Authors

- **Njuguna Mureithi** - _Initial work_ - [Njuguna Mureithi](https://github.com/geofmureithi)

See also the list of [contributors](https://github.com/geofmureithi/apalis/contributors) who participated in this project.

## License

This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details

[`tower::Service`]: https://docs.rs/tower/latest/tower/trait.Service.html
[tower]: https://crates.io/crates/tower
[`actix`]: https://crates.io/crates/actix