Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/sorentwo/kiq

:postbox: Robust job queue powered by GenStage and Redis
https://github.com/sorentwo/kiq

background-jobs elixir genstage redis workers

Last synced: 13 days ago
JSON representation

:postbox: Robust job queue powered by GenStage and Redis

Awesome Lists containing this project

README 

        
# Kiq



[![Build Status](https://travis-ci.org/sorentwo/kiq.svg?branch=master)](https://travis-ci.org/sorentwo/kiq)

[![Hex.pm Version](http://img.shields.io/hexpm/v/kiq.svg)](https://hex.pm/packages/kiq)

[![Hex.pm Downloads](https://img.shields.io/hexpm/dt/kiq.svg)](https://hex.pm/packages/kiq)

[![Hex Docs](http://img.shields.io/badge/hex.pm-docs-green.svg?style=flat)](https://hexdocs.pm/kiq)



Kiq is a robust and extensible job processing queue that aims for compatibility

with [Sidekiq][sk], [Sidekiq Pro][skp] and [Sidekiq Enterprise][ske].



Job queuing, processing and reporting are all built on [GenStage][genst]. That means

maximum parallelism with the safety of backpressure as jobs are processed.



[genst]: https://github.com/elixir-lang/gen_stage



### Why Kiq?



Many features and architectural choices in Kiq were drawn from existing Elixir

job processing packages like [Exq][exq], [Verk][verk] and [EctoJob][ej]. Each

of those packages are great and have varying strenghts, but they lacked seamless

interop with Sidekiq Pro or Sidekiq Enterprise jobs.



Sidekiq Pro and Enterprise are amazing pieces of commercial software and worth

every cent—your organization should buy them! Kiq is intended as a _bridge_ for

your team to _interop_ between Ruby and Elixir. As an organization embraces

Elixir it becomes necessary to run some background jobs in Elixir, and it must

be just as reliable as when jobs were ran through Sidekiq.



[sk]: https://sidekiq.org/

[skp]: https://sidekiq.org/products/pro.html

[ske]: https://sidekiq.org/products/enterprise.html

[exq]: https://github.com/akira/exq

[verk]: https://github.com/edgurgel/verk

[ej]: https://github.com/mbuhot/ecto_job



### Sidekiq Pro & Enterprise Comaptible Feature Set



Kiq's feature set includes many marquee offerings from Sidekiq, Sidekiq Pro and

Sidekiq Enterprise—plus some additional niceties made possible by running on the

BEAM. Here is a table highlighting the Kiq's features compared to the various

Sidekiq versions:



| Feature            | Kiq         | Sidekiq    | Sidekiq Pro | Sidekiq Ent |

| ------------------ | ----------- | ---------- | ----------- | ----------- |

| Max Size Queues    | ✅          | ❌         | ❌          | ❌          |

| Structured Logging | ✅          | ❌         | ❌          | ❌          |

| Scheduled Jobs     | ✅          | ✅         | ✅          | ✅          |

| Error Handling     | ✅          | ✅         | ✅          | ✅          |

| Expiring Jobs      | ✅          | ❌         | ✅          | ✅          |

| Worker Metrics     | ✅          | ❌         | ✅          | ✅          |

| Reliable Client    | ✅          | ❌         | ✅          | ✅          |

| Reliable Server    | ✅          | ❌         | ✅          | ✅          |

| Rolling Restarts   | ✅          | ❌         | ❌          | ✅          |

| Periodic Jobs      | ✅          | ❌         | ❌          | ✅          |

| Unique Jobs        | ✅          | ❌         | ❌          | ✅          |

| Leader Election    | ✅          | ❌         | ❌          | ✅          |

| Multi Process      | ✅          | ❌         | ❌          | ✅          |

| Web UI †           | ❌          | ✅         | ✅          | ✅          |

| Batch Jobs ‡       | ❌          | ❌         | ✅          | ✅          |

| Encryption ‡       | ❌          | ❌         | ❌          | ✅          |

| Rate Limiting ‡    | ❌          | ❌         | ❌          | ✅          |



* † Kiq relies on Sidekiq's Web UI

* ‡ Planned, but not implemented yet



If a feature isn't supported or planned it is _probably_ for one of these

reasons:



1. We get it for free on the BEAM and it isn't necessary, i.e. (safe shutdown,

   multi-process, rolling restarts)

3. We enable developers to use custom reporters to do it themselves (stats,

   error reporting)



### Design Decisions



* Avoid global and compile time configuration. All configuration can be defined

  programatically, eliminating the need for hacks like `{:system, "REDIS_URL"}`.

* Not an application, it is included in your application's supervision tree

* Testing focused, provide helpers and modes to aid testing

* Extensible job handling via GenStage consumers

* Simplified worker definitions to ease job definition and pipelining



[ent]: https://sidekiq.org/products/enterprise.html



## Installation



Add `kiq` to your list of dependencies in `mix.exs`:



```elixir

def deps do

  [

    {:kiq, "~> 0.7"}

  ]

end

```



Then run `mix deps.get` to install the dependency.



Finally, add the supervisor to your application's supervision tree:



```elixir

{MyApp.Kiq, []}

```



Kiq itself is not an application and must be started within your application's

supervision tree. All of your application's configuration and custom methods

should be put into the supervisor.



## Usage



Kiq isn't an application that must be started. Similarly to Ecto, you define

one or more Kiq modules within your application. This allows multiple

supervision trees with entirely different configurations.



Run the generator to define a Kiq supervisor for your application:



```bash

mix kiq.gen.supervisor MyApp.Kiq

```



Include the module in your application's supervision tree:



```elixir

defmodule MyApp.Application do

  @moduledoc false



  use Application



  alias MyApp.{Endpoint, Kiq, Repo}



  def start(_type, _args) do

    children = [

      {Repo, []},

      {Endpoint, []},

      {Kiq, []}

    ]



    Supervisor.start_link(children, strategy: :one_for_one, name: MyApp.Supervisor)

  end

end

```



With the supervision tree in place you are ready to start creating workers! The

simplest way to create a worker is through the generator:



```bash

mix kiq.gen.worker MyApp.Workers.Business

```



That will define a worker with a `perform/1` function where all the magic will

happen.



See `mix help kiq.gen.worker` for additional options.



Check the [hexdocs][hd] for additional details, configuration options, how to

test, defining workers and custom reporters.



[hd]: https://hexdocs.pm/kiq



## Benchmarks



Kiq has a set of benchmarks to track the performance of important operations.

Benchmarks are ran using the [Benchee][benchee] library and require Redis to be

running.



To run all benchmarks:



```bash

mix run bench/bench_helper.exs

```



[benchee]: https://github.com/PragTob/benchee



## Contributing



Clone the repository and run `$ mix test` to make sure everything is working. For

tests to pass, you must have a Redis server running on `localhost`, port `6379`,

database `3`. You can configure a different host, port and database by setting

the `REDIS_URL` environment variable before testing.



Note that tests will wipe the the configured database on the Redis server

multiple times while testing. By default database 3 is used for testing.



## License



Kiq is released under the MIT license. See the [LICENSE](LICENSE.txt).