{"id":18383893,"url":"https://github.com/phillipjhl/white_rabbit","last_synced_at":"2025-04-11T23:27:54.719Z","repository":{"id":65145748,"uuid":"564514099","full_name":"phillipjhl/white_rabbit","owner":"phillipjhl","description":"Elixir library that uses the AMQP to provide behaviors that dynamically supervises all connections, channels, consuming, producing, and exchanging of RabbitMQ messages. Also provides an api to send RPC messages to other distributed elixir nodes via reply_to queues.","archived":false,"fork":false,"pushed_at":"2023-01-17T20:32:17.000Z","size":915,"stargazers_count":2,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-16T00:12:44.012Z","etag":null,"topics":["amqp","autorecovery","behaviour","consumer","dynamic-supervisor","elixir","genserver","producer","protocol","rabbit","rabbitmq","rpc","telemetry"],"latest_commit_sha":null,"homepage":"https://hexdocs.pm/white_rabbit/","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/phillipjhl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-11-10T22:03:10.000Z","updated_at":"2024-11-17T21:35:40.000Z","dependencies_parsed_at":"2023-02-10T12:00:24.901Z","dependency_job_id":null,"html_url":"https://github.com/phillipjhl/white_rabbit","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phillipjhl%2Fwhite_rabbit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phillipjhl%2Fwhite_rabbit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phillipjhl%2Fwhite_rabbit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/phillipjhl%2Fwhite_rabbit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/phillipjhl","download_url":"https://codeload.github.com/phillipjhl/white_rabbit/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248494228,"owners_count":21113402,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amqp","autorecovery","behaviour","consumer","dynamic-supervisor","elixir","genserver","producer","protocol","rabbit","rabbitmq","rpc","telemetry"],"created_at":"2024-11-06T01:12:54.434Z","updated_at":"2025-04-11T23:27:54.697Z","avatar_url":"https://github.com/phillipjhl.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"# White Rabbit\n\n![hexdocs.pm/white_rabbit](https://img.shields.io/hexpm/v/white_rabbit)\n\nRabbitMQ Elixir library to handle all consuming, producing, and exchanging of RabbitMQ messages.\n\nThis lightweight library makes use of the AMQP 0-9-1 protocol. Use WhiteRabbit to help with scaling and processing of your RabbitMQ message queues. \n\nWhether you need just connection and channel recovery with running processes, or want to take advantage of the RPC call architecture across independent distributed OTP applications with the benefit of having your messages being sent to your RabbitMQ's queues; this library will build upon whatever is required for your application. \n\nFeatures:\n\n- RPC call architecture between apps using RabbitMQ and reply_to key routing.\n- Auto recovery and restarts of connections, their channels, and the consumers using them\n- Startup and runtime support for Consumer GenServer processes. Start them at runtime using Elixir's DynamicSupervisors.\n- Telemetry event emission on publishes, acks, and rejects\n\nIf this isn't exactly what you are looking for, I recommend the well-known elixir library [broadway](https://github.com/dashbitco/broadway). Even though that is intended for multi-stage data ingest using GenStages and has more batch-level processing in mind, it has similar features that could fit your needs.\n\n## v0.1.0 Topology\n\n![Topology Flow](assets/WhiteRabbit_Topology.svg)\n\n# Implementing\n\n## Add To Dependencies\n\n```elixir\n  defp deps do\n    [\n      {:white_rabbit, \"~\u003e 0.2.0\"},\n    ]\n  end\n```\n\n## Use As A Behavior\n\n```elixir\n# In MyApp\n\ndefmodule MyApp.WhiteRabbit do\n  use WhiteRabbit\n\n  def start_link(_opts) do\n    WhiteRabbit.start_link(__MODULE__, name: __MODULE__)\n  end\n\n  # Callbacks below\nend\n```\n\n```elixir\n# In Main Application Supervisor\n\nchildren = [\n  # other fun stuff...\n  {MyApp.WhiteRabbit, []}\n]\n\nSupervisor.start_link(children, opts)\n```\n\n### Define Processor Modules\n\n`WhiteRabbit.Processor` Behaviour is used to define how a `WhiteRabbit.Consumer` Genserver will process the messages that it receives once it is registered to a RabbitMQ queue.\n\nThe `consume_payload/2` callback is called immediately after a message is received. It should return a `{:ok, AMQP.Basic.delivery_tag()}` tuple if successful or a `{:error, {AMQP.Basic.delivery_tag(), Keyword.t()}}` if not successful so the Consumer can properly send an 'ack' or 'reject' for the message and then let RabbitMQ broker can handle it appropriately (requeue, send to dead-letter-queue, etc.)\n\nSee `WhiteRabbit.Processor` for more on the behaviour used.\n\nExample:\n\n```elixir\ndefmodule AppOne.TestJsonProcessor do\n  @behaviour WhiteRabbit.Processor\n\n  require Logger\n\n  @impl true\n  def consume_payload(\n        payload,\n        %{delivery_tag: tag, redelivered: redelivered, content_type: content_type} = meta\n      ) do\n    Logger.debug(\"Processing message #{tag} with #{__MODULE__}\")\n\n    case content_type do\n      \"application/json\" -\u003e\n        {:ok, _json} = Jason.decode(payload)\n\n        # If successful, send back `{:ok, AMQP.Basic.delivery_tag()}`\n        {:ok, tag}\n\n      _ -\u003e\n        {:error, {tag, [requeue: false]}}\n    end\n  rescue\n    exception -\u003e\n      # Requeue unless it's a redelivered message.\n      # This means we will retry consuming a message once in case of exception\n      # before we give up and have it moved to the error queue\n      Logger.warn(\"Error consuming message: #{tag} #{inspect(exception)}\")\n      {:error, {tag, [requeue: not redelivered]}}\n  end\nend\n```\n\n### Define Optional Startup Consumers\n\nSee `WhiteRabbit.Consumer` for more info.\n\n```elixir\n@impl true\ndef get_startup_consumers do\n  [\n    {WhiteRabbit.Consumer, %WhiteRabbit.Consumer{\n      connection_name: :appone_connection,\n      name: \"AppOne.JsonConsumer\",\n      exchange: \"json_test_exchange\",\n      queue: \"json_test_queue\",\n      processor: %WhiteRabbit.Processor.Config{module: AppOne.TestJsonProcessor}\n      }\n    }\n  ]\nend\n```\n\n### Full Example Module\n\n```elixir\ndefmodule AppOne.WhiteRabbit do\n  use WhiteRabbit\n\n  alias WhiteRabbit.{Connection, Core}\n\n  # Define list of connections for this WhiteRabbit instance\n  def get_connections do\n    [\n      %Connection{\n        connection_name: :appone_connection,\n        conn_opts: [url: \"amqp://user:pass@localhost:5673/dev\"],\n        channels: [\n          %{\n            name: :appone_consumer_channel\n          },\n          %{\n            name: :appone_producer_channel\n          }\n        ]\n      },\n      %Connection{\n        connection_name: :appone_rpc_connection,\n        conn_opts: [url: \"amqp://user:pass@localhost:5673/dev\"],\n        channels: [\n          %{\n            name: :appone_rpc_consumer_channel_1\n          },\n          %{\n            name: :appone_rpc_consumer_channel_2\n          }\n        ]\n      }\n    ]\n  end\n\n  # Use callback spec to return %WhiteRabbit.RPC.Config{} struct\n  @impl true\n  def get_rpc_config do\n    %WhiteRabbit.RPC.Config{\n      service_name: :appone,\n      connection_name: :appone_rpc_connection\n    }\n  end\n\n  # Set optional Consumer process to start consuming at app start-up\n  @impl true\n  def get_startup_consumers do\n    [\n      {WhiteRabbit.Consumer, %WhiteRabbit.Consumer{\n          connection_name: :appone_connection,\n          name: \"AppOne.JsonConsumer\",\n          exchange: \"json_test_exchange\",\n          queue: \"json_test_queue\",\n          processor: %WhiteRabbit.Processor.Config{module: AppOne.TestJsonProcessor}\n        }\n      }\n    ]\n  end\n\n  def start_link(_opts) do\n    rpc_enabled = if Mix.env() !== :test, do: true, else: false\n\n    # Start the WhiteRabbit Hole\n    WhiteRabbit.start_link(__MODULE__,\n      name: __MODULE__,\n      connections: connections(),\n      rpc_enabled: rpc_enabled,\n      rpc_config: get_rpc_config()\n    )\n  end\nend\n\n```\n\n## Testing\n\n### Starting Dynamic Consumers\n\nThe processor module needs to exist before trying to register a consumer for it.\n\nSee `WhiteRabbit.Processor` module for more info on that behaviour\n\n```elixir\nappone_config = %WhiteRabbit.Consumer{\n  owner_module: AppOne.WhiteRabbit,\n  connection_name: :appone_connection,\n  name: \"AppOne.TestJsonProcessor\",\n  exchange: \"json_test_exchange\",\n  queue: \"json_test_queue\",\n  processor: %WhiteRabbit.Processor.Config{module: AppOne.TestJsonProcessor}\n}\n\napptwo_config = %WhiteRabbit.Consumer{\n  owner_module: AppTwo.WhiteRabbit,\n  connection_name: :apptwo_connection,\n  name: \"AppTwo.TestJsonProcessor\",\n  exchange: \"apptwo_json_test_exchange\",\n  queue: \"apptwo_json_test_queue\",\n  processor: %WhiteRabbit.Processor.Config{module: AppTwo.TestJsonProcessor}\n}\n\nAppOne.WhiteRabbit.start_dynamic_consumers(appone_config, 3)\nAppTwo.WhiteRabbit.start_dynamic_consumers(apptwo_config, 3)\n\n# or specifying the owner module\n\nWhiteRabbit.start_dynamic_consumers(appone_config, 3, AppOne.WhiteRabbit)\nWhiteRabbit.start_dynamic_consumers(apptwo_config, 3, AppTwo.WhiteRabbit)\n```\n\n### Publish With Producer Module\n\nSee `WhiteRabbit.Producer`\n\nIf using the Producer as a behavior there is a providing `publish/5` that you can just provide the connection_name so it will use the correct connection pool and use a channel under that already opened connection.\n\nThis method will make sure the channel and connection used are from the calling application which will be under the correct supervision tree.\n\nThis is the prefered method especially if there are serveral applications with `WhiteRabbit` modules under an umbrella application.\n\n```elixir\ndefmodule AppOne.Producer.Json do\n  use WhiteRabbit.Producer\n\n  def send_json(payload) do\n    publish(:appone_connection, \"json_test_exchange\", \"test_json\", payload,\n        content_type: \"application/json\",\n        persistent: true\n      )\n  end\nend\n```\n\nIf you know the channel registry name and the connection name of the pool to use then you can use the `publish/5` function with a tuple as the first argument.\n\n```elixir\nWhiteRabbit.Producer.publish({:appone_connection, AppOne.WhiteRabbit.ChannelRegistry}, \"test_exchange\", \"test_route\", \"hello there\", persistent: true)\n```\n\nIf there is an already connected channel you can use the `publish/5`.\n\nExample:\n\n```elixir\nchannel = %AMQP.Channel{}\n\nWhiteRabbit.Producer.publish(channel, \"test_exchange\", \"test_route\", \"hello there\", persistent: true)\n```\n\n### Test Publish to Exchange\n\n```elixir\nWhiteRabbit.Core.test_publish(100, \"json_test_exchange\", \"test_json\", %{hello: \"there\"})\nWhiteRabbit.Core.test_publish(100, \"apptwo_json_test_exchange\", \"test_json\", %{hello: \"there\"})\n```\n\n### RPC Calls\n\nSee `WhiteRabbit.RPC` for more information on RPC calls and message handling.\n\nUsing the RPC config shown above, a process can call the function `WhiteRabbit.RPC.call/4` to make an RPC call to the service and get a response back from the server and have it map back correctly to the calling process.\n\nIf using as a behavior, then a `rpc_call/3` function will be provided that accepts the service to call, mfa tuple, and options.\n\nExample:\n\n```elixir\niex\u003e AppFour.WhiteRabbit.rpc_call(:appone, {AppOne.Utils, :get_versions, []})\n{:ok,\n [\n   %{\"name\" =\u003e \"appone\", \"version\" =\u003e \"3.5.0\"},\n   %{\"name\" =\u003e \"custom_lib_1\", \"version\" =\u003e \"1.3.0\"},\n   %{\"name\" =\u003e \"processing_lib\", \"version\" =\u003e \"2.6.0\"}\n ]}\n```\n\n# To Generate ExDocs\n\n```\n$ mix docs\n```\n\n# Copyright and License\n\nCopyright (c) 2021, Phillip Langland\n\nWhiteRabbit source code is licensed under the [MIT License](./LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphillipjhl%2Fwhite_rabbit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fphillipjhl%2Fwhite_rabbit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphillipjhl%2Fwhite_rabbit/lists"}