{"id":16415421,"url":"https://github.com/sorentwo/kiq","last_synced_at":"2026-05-15T03:30:17.546Z","repository":{"id":32805646,"uuid":"138225519","full_name":"sorentwo/kiq","owner":"sorentwo","description":":postbox: Robust job queue powered by GenStage and Redis","archived":false,"fork":false,"pushed_at":"2025-02-11T12:02:17.000Z","size":365,"stargazers_count":75,"open_issues_count":1,"forks_count":6,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-16T03:27:08.234Z","etag":null,"topics":["background-jobs","elixir","genstage","redis","workers"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sorentwo.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-06-21T21:54:29.000Z","updated_at":"2025-02-11T12:02:20.000Z","dependencies_parsed_at":"2023-02-18T01:15:43.185Z","dependency_job_id":"94ca3d6e-a9b8-492e-9c3e-d74248c3242b","html_url":"https://github.com/sorentwo/kiq","commit_stats":{"total_commits":258,"total_committers":4,"mean_commits":64.5,"dds":0.4534883720930233,"last_synced_commit":"ea5a622554bf21914eb5186d756b6cfd94d3735d"},"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sorentwo%2Fkiq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sorentwo%2Fkiq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sorentwo%2Fkiq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sorentwo%2Fkiq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sorentwo","download_url":"https://codeload.github.com/sorentwo/kiq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240118411,"owners_count":19750488,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["background-jobs","elixir","genstage","redis","workers"],"created_at":"2024-10-11T07:05:39.775Z","updated_at":"2026-05-15T03:30:17.476Z","avatar_url":"https://github.com/sorentwo.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kiq\n\n[![Build Status](https://travis-ci.org/sorentwo/kiq.svg?branch=master)](https://travis-ci.org/sorentwo/kiq)\n[![Hex.pm Version](http://img.shields.io/hexpm/v/kiq.svg)](https://hex.pm/packages/kiq)\n[![Hex.pm Downloads](https://img.shields.io/hexpm/dt/kiq.svg)](https://hex.pm/packages/kiq)\n[![Hex Docs](http://img.shields.io/badge/hex.pm-docs-green.svg?style=flat)](https://hexdocs.pm/kiq)\n\nKiq is a robust and extensible job processing queue that aims for compatibility\nwith [Sidekiq][sk], [Sidekiq Pro][skp] and [Sidekiq Enterprise][ske].\n\nJob queuing, processing and reporting are all built on [GenStage][genst]. That means\nmaximum parallelism with the safety of backpressure as jobs are processed.\n\n[genst]: https://github.com/elixir-lang/gen_stage\n\n### Why Kiq?\n\nMany features and architectural choices in Kiq were drawn from existing Elixir\njob processing packages like [Exq][exq], [Verk][verk] and [EctoJob][ej]. Each\nof those packages are great and have varying strenghts, but they lacked seamless\ninterop with Sidekiq Pro or Sidekiq Enterprise jobs.\n\nSidekiq Pro and Enterprise are amazing pieces of commercial software and worth\nevery cent—your organization should buy them! Kiq is intended as a _bridge_ for\nyour team to _interop_ between Ruby and Elixir. As an organization embraces\nElixir it becomes necessary to run some background jobs in Elixir, and it must\nbe just as reliable as when jobs were ran through Sidekiq.\n\n[sk]: https://sidekiq.org/\n[skp]: https://sidekiq.org/products/pro.html\n[ske]: https://sidekiq.org/products/enterprise.html\n[exq]: https://github.com/akira/exq\n[verk]: https://github.com/edgurgel/verk\n[ej]: https://github.com/mbuhot/ecto_job\n\n### Sidekiq Pro \u0026 Enterprise Comaptible Feature Set\n\nKiq's feature set includes many marquee offerings from Sidekiq, Sidekiq Pro and\nSidekiq Enterprise—plus some additional niceties made possible by running on the\nBEAM. Here is a table highlighting the Kiq's features compared to the various\nSidekiq versions:\n\n| Feature            | Kiq         | Sidekiq    | Sidekiq Pro | Sidekiq Ent |\n| ------------------ | ----------- | ---------- | ----------- | ----------- |\n| Max Size Queues    | ✅          | ❌         | ❌          | ❌          |\n| Structured Logging | ✅          | ❌         | ❌          | ❌          |\n| Scheduled Jobs     | ✅          | ✅         | ✅          | ✅          |\n| Error Handling     | ✅          | ✅         | ✅          | ✅          |\n| Expiring Jobs      | ✅          | ❌         | ✅          | ✅          |\n| Worker Metrics     | ✅          | ❌         | ✅          | ✅          |\n| Reliable Client    | ✅          | ❌         | ✅          | ✅          |\n| Reliable Server    | ✅          | ❌         | ✅          | ✅          |\n| Rolling Restarts   | ✅          | ❌         | ❌          | ✅          |\n| Periodic Jobs      | ✅          | ❌         | ❌          | ✅          |\n| Unique Jobs        | ✅          | ❌         | ❌          | ✅          |\n| Leader Election    | ✅          | ❌         | ❌          | ✅          |\n| Multi Process      | ✅          | ❌         | ❌          | ✅          |\n| Web UI †           | ❌          | ✅         | ✅          | ✅          |\n| Batch Jobs ‡       | ❌          | ❌         | ✅          | ✅          |\n| Encryption ‡       | ❌          | ❌         | ❌          | ✅          |\n| Rate Limiting ‡    | ❌          | ❌         | ❌          | ✅          |\n\n* † Kiq relies on Sidekiq's Web UI\n* ‡ Planned, but not implemented yet\n\nIf a feature isn't supported or planned it is _probably_ for one of these\nreasons:\n\n1. We get it for free on the BEAM and it isn't necessary, i.e. (safe shutdown,\n   multi-process, rolling restarts)\n3. We enable developers to use custom reporters to do it themselves (stats,\n   error reporting)\n\n### Design Decisions\n\n* Avoid global and compile time configuration. All configuration can be defined\n  programatically, eliminating the need for hacks like `{:system, \"REDIS_URL\"}`.\n* Not an application, it is included in your application's supervision tree\n* Testing focused, provide helpers and modes to aid testing\n* Extensible job handling via GenStage consumers\n* Simplified worker definitions to ease job definition and pipelining\n\n[ent]: https://sidekiq.org/products/enterprise.html\n\n## Installation\n\nAdd `kiq` to your list of dependencies in `mix.exs`:\n\n```elixir\ndef deps do\n  [\n    {:kiq, \"~\u003e 0.7\"}\n  ]\nend\n```\n\nThen run `mix deps.get` to install the dependency.\n\n\nFinally, add the supervisor to your application's supervision tree:\n\n```elixir\n{MyApp.Kiq, []}\n```\n\nKiq itself is not an application and must be started within your application's\nsupervision tree. All of your application's configuration and custom methods\nshould be put into the supervisor.\n\n## Usage\n\nKiq isn't an application that must be started. Similarly to Ecto, you define\none or more Kiq modules within your application. This allows multiple\nsupervision trees with entirely different configurations.\n\nRun the generator to define a Kiq supervisor for your application:\n\n```bash\nmix kiq.gen.supervisor MyApp.Kiq\n```\n\nInclude the module in your application's supervision tree:\n\n```elixir\ndefmodule MyApp.Application do\n  @moduledoc false\n\n  use Application\n\n  alias MyApp.{Endpoint, Kiq, Repo}\n\n  def start(_type, _args) do\n    children = [\n      {Repo, []},\n      {Endpoint, []},\n      {Kiq, []}\n    ]\n\n    Supervisor.start_link(children, strategy: :one_for_one, name: MyApp.Supervisor)\n  end\nend\n```\n\nWith the supervision tree in place you are ready to start creating workers! The\nsimplest way to create a worker is through the generator:\n\n```bash\nmix kiq.gen.worker MyApp.Workers.Business\n```\n\nThat will define a worker with a `perform/1` function where all the magic will\nhappen.\n\nSee `mix help kiq.gen.worker` for additional options.\n\nCheck the [hexdocs][hd] for additional details, configuration options, how to\ntest, defining workers and custom reporters.\n\n[hd]: https://hexdocs.pm/kiq\n\n## Benchmarks\n\nKiq has a set of benchmarks to track the performance of important operations.\nBenchmarks are ran using the [Benchee][benchee] library and require Redis to be\nrunning.\n\nTo run all benchmarks:\n\n```bash\nmix run bench/bench_helper.exs\n```\n\n[benchee]: https://github.com/PragTob/benchee\n\n## Contributing\n\nClone the repository and run `$ mix test` to make sure everything is working. For\ntests to pass, you must have a Redis server running on `localhost`, port `6379`,\ndatabase `3`. You can configure a different host, port and database by setting\nthe `REDIS_URL` environment variable before testing.\n\nNote that tests will wipe the the configured database on the Redis server\nmultiple times while testing. By default database 3 is used for testing.\n\n## License\n\nKiq is released under the MIT license. See the [LICENSE](LICENSE.txt).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsorentwo%2Fkiq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsorentwo%2Fkiq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsorentwo%2Fkiq/lists"}