{"id":15176431,"url":"https://github.com/monimesl/pulserl","last_synced_at":"2025-10-26T11:31:33.092Z","repository":{"id":48978236,"uuid":"225132360","full_name":"monimesl/pulserl","owner":"monimesl","description":"Apache Pulsar client library for Erlang/Elixir","archived":false,"fork":false,"pushed_at":"2024-08-13T22:23:51.000Z","size":391,"stargazers_count":15,"open_issues_count":2,"forks_count":11,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-01-10T17:36:09.857Z","etag":null,"topics":["data-processing","elixir","erlang","pulsar","pulsar-client"],"latest_commit_sha":null,"homepage":null,"language":"Erlang","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/monimesl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-12-01T08:56:48.000Z","updated_at":"2024-10-23T23:13:07.000Z","dependencies_parsed_at":"2024-08-14T01:14:25.830Z","dependency_job_id":"3adcf781-745d-424b-9f40-f2d3315f0b50","html_url":"https://github.com/monimesl/pulserl","commit_stats":null,"previous_names":["monimesl/pulserl","skulup/pulserl"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monimesl%2Fpulserl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monimesl%2Fpulserl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monimesl%2Fpulserl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monimesl%2Fpulserl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/monimesl","download_url":"https://codeload.github.com/monimesl/pulserl/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237995679,"owners_count":19399176,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-processing","elixir","erlang","pulsar","pulsar-client"],"created_at":"2024-09-27T13:04:11.126Z","updated_at":"2025-10-26T11:31:32.642Z","avatar_url":"https://github.com/monimesl.png","language":"Erlang","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.com/skulup/pulserl.svg?branch=master)](https://travis-ci.com/skulup/pulserl)\n[![Codacy Badge](https://api.codacy.com/project/badge/Grade/731f6fe6fa9442ce8b8d2b994a121d93)](https://app.codacy.com/gh/skulup/pulserl?utm_source=github.com\u0026utm_medium=referral\u0026utm_content=skulup/pulserl\u0026utm_campaign=Badge_Grade_Dashboard)\n[![Language](https://img.shields.io/badge/Language-Erlang-b83998.svg)](https://www.erlang.org/)\n[![LICENSE](https://img.shields.io/badge/License-Apache%202-blue.svg)](https://github.com/skulup/pulserl/blob/master/LICENSE)\n# Pulserl\n#### An Apache Pulsar client for Erlang/Elixir\n__Version:__ 0.1.3\n\nPulserl is an Erlang client for the Apache Pulsar Pub/Sub system with both producer and consumer\nimplementations. It requires a version __2.0+__ of Apache Pulsar and __18.0+__ of Erlang.\nPulserl uses the [binary protocol](http://pulsar.apache.org/docs/en/develop-binary-protocol)\nto interact with the Pulsar brokers and exposes a very simple API.\n## Quick Examples\n\nThe examples assume you have a running Pulsar broker at `localhost:6650`, a topic called `test-topic` (can be partitioned or not) and `rebar3` installed.\n\n_Note: Pulserl uses `pulserl` and `Shared` as the default subscription name and type.\n So, if that subscription (not the consumer) under the topic `test-topic` does not exists, we make sure in this example to create it first by creating\n the consumer before producing any message to the topic._\n\nFetch, compile and start the erlang shell.\n```bash\n  git clone https://github.com/skulup/pulserl.git,\n  cd pulserl\n  rebar3 shell\n```\n\nIn the Erlang shell\n```erlang\n  rr(pulserl).  %% load the api records\n\n  %% A demo function to log the value of consumed messages\n  %% that will be produced blow.\n  pulserl:start_consumption_in_background(\"test-topic\").\n\n  %% Asycnhrounous produce\n  Promise = pulserl:produce(\"test-topic\", \"Asycn produce message\").\n  pulserl:await(Promise).  %% Wait broker ack\n  #messageId{ledger_id = 172,entry_id = 7,\n             topic = \u003c\u003c\"persistent://public/default/test-topic\"\u003e\u003e,\n             partition = -1,batch = undefined}\n\n  %% Asycnhrounous produce. Response notification is via callback (fun/1)\n  pulserl:produce(\"test-topic\", \"Hello\", fun(Res) -\u003e io:format(\"Response: ~p~n\", [Res]) end).\n\n  %% Synchronous produce\n  pulserl:sync_produce(\"test-topic\", \"Sync produce message\").\n  #messageId{ledger_id = 176,entry_id = 11,\n             topic = \u003c\u003c\"persistent://public/default/test-topic\"\u003e\u003e,\n             partition = -1,batch = undefined}\n\n```\n\n## Feature Matrix\n\n - [x] [Basic Producer](http://pulsar.apache.org/docs/en/concepts-messaging/#producers)\n - [x] [Basic Consumer](http://pulsar.apache.org/docs/en/concepts-messaging/#consumers)\n - [x] [Partitioned topics](http://pulsar.apache.org/docs/en/concepts-messaging/#partitioned-topics)\n - [x] [Batching](http://pulsar.apache.org/docs/en/concepts-messaging/#batching)\n - [ ] [Compression](http://pulsar.apache.org/docs/en/concepts-messaging/#compression)\n - [x] [TLS](https://pulsar.apache.org/docs/en/security-tls-transport/#tls-overview)\n - [ ] [Authentication (token, tls)](https://pulsar.apache.org/docs/en/security-overview/)\n - [ ] [Reader API](https://pulsar.apache.org/docs/en/concepts-clients/#reader-interface)\n - [x] [Proxy Support (for Kubernetes)](http://pulsar.apache.org/docs/en/concepts-architecture-overview/#pulsar-proxy)\n - [x] [Effectively-Once](https://pulsar.apache.org/docs/en/concepts-messaging/#deduplication-and-effectively-once-semantics)\n - [ ] [Schema](https://pulsar.apache.org/docs/en/schema-get-started/)\n - [x] Consumer seek\n - [ ] [Multi-topics consumer](https://pulsar.apache.org/docs/en/concepts-messaging/#multi-topic-subscriptions)\n - [ ] [Topics regex consumer](https://github.com/apache/pulsar/wiki/PIP-13:-Subscribe-to-topics-represented-by-regular-expressions)\n - [ ] [Compacted topics](https://pulsar.apache.org/docs/en/concepts-topic-compaction/#compaction)\n - [x] User defined properties producer/consumer\n - [ ] Reader hasMessageAvailable\n - [ ] [Hostname verification](https://pulsar.apache.org/docs/en/2.3.1/security-tls-transport/#hostname-verification)\n - [ ] [Multi Hosts Service Url support](https://pulsar.apache.org/docs/en/admin-api-overview/#java-admin-client)\n - [x] [Key_shared](https://pulsar.apache.org/docs/en/concepts-messaging/#key_shared)\n - [ ] key based batcher (didn't find a documentation) ?\n - [x] [Negative Acknowledge](https://pulsar.apache.org/docs/en/concepts-messaging/#negative-acknowledgement)\n - [x] [Delayed Delivery Messages](https://pulsar.apache.org/docs/en/concepts-messaging/#delayed-message-delivery)\n - [x] [Dead Letter Policy](https://pulsar.apache.org/docs/en/concepts-messaging/#dead-letter-topic)\n - [ ] [Interceptors](https://github.com/apache/pulsar/wiki/PIP-23:-Message-Tracing-By-Interceptors)\n\n _Thanks to Sabudaye for [this information](https://github.com/skulup/pulserl/issues/2#issuecomment-616463542)_\n\n\n## Installation\n [Pulserl is available in Hex](https://hex.pm/packages/pulserl) for easy installation by added it to your project dependencies.\n\nIn your Erlang project's `rebar.config`\n ```erlang\n{deps, [\n   {pulserl, \"\u003clatest-version\u003e\"}\n]}\n```\n\nIn your Elixir project's `mix.exs`\n ```elixir\ndef deps do\n  [\n    {:pulserl, \"~\u003e \u003clatest-version\u003e\"}\n  ]\nend\n```\n\n## API Usage\n\n### Client Setup\n  In pulserl, the client as of now (for API simplicity) is a singleton (local registered `gen_server`)\n  and can be created during startup by the `application controller` or on demand at a later time.\n  The client has the responsibility of creating the TCP connections,\n  maintaining the connection pool, and ensures these connections are\n  maximally used by the producers and consumers. The client is also responsible\n  for querying metadata needed to initialize a producer or consumer; it does this\n  by creating a metadata socket during initialization by using the provided configurations.\n\n  #### Automatic client startup\n   You can configure the client that will be auto-started by providing\n   the following configuration for `pulserl` in your `sys.config` file.\n```erlang\n[\n  {pulserl, [\n    {autostart, true} %% If true, the client will be created on startup. Default is true.\n    %% The TCP connect timeout in milliseconds. Default is 30000.\n    , {connect_timeout_ms, 30000}\n    %% The maximum connections to each broker the client should create.\n    %% Default is 1. Increasing this may improve I/O throughput\n    , {max_connections_per_broker, 1}\n    %% The underlying TCP socket options.\n    %% https://erlang.org/doc/man/gen_tcp.html#type-connect_option\n    , {socket_options, [{nodelay, true}]}\n    %% The service url. Default is the non TLS url: \"pulsar://${hostname}:6650\"\n    , {service_url, \"pulsar+ssl://localhost:6651/\"}\n    %% The trust certificate file path. Required only if the TLS service url is used.\n    %% See http://pulsar.apache.org/docs/en/security-tls-transport/\n    , {tls_trust_certs_file, \"/path/to/cacert.pem\"}\n  ]}\n].\n```\n  #### On demand client startup\n  The `pulserl:start_client/1,2` API can be used to start the pulserl client when needed.\n```erlang\n ServiceUrl = \"pulsar+ssl://localhost:6651/\",\n Config = #clientConfig{\n             connect_timeout_ms = 30000,\n             max_connections_per_broker = 1,\n             socket_options = [{nodelay, true}],\n             tls_trust_certs_file = \"/path/to/cacert.pem\"\n           },\n ok = pulserl:start_client(ServiceUrl, ClientConfig).\n```\n\n### Producer\nPulserl creates a `gen_server` process per topic. For a topic of `n` partition, it creates\na parent producer under the `pulserl_producer_sup` supervision tree which in turn `start_link`\nand manage `n` child producers. The parent producer serves as a facade to the internal producers.\nThe parent monitor the child processes (internal partitioned producers) for resilience,\nroute client calls to one of the child processes using different\n[routing modes](https://pulsar.apache.org/docs/en/concepts-messaging/#routing-modes).\nA producer during initialization is assigned a connection by the client based on its topic metadata.\nA producer uses a queueing mechanism on message sending.\nEach send is internally a `gen_server.call/2` to the producer process. The caller's reference is\nadded to a queue and replied immediately with `ok.` This initial early reply frees up the caller to do\nother tasks if the response is not needed immediately. Internally if message send is trigger, i.e\nwhen batching is not enable or batching enabled, but a batch send is triggered, the producer\nasynchronously send (`gen_sever.cast/2`) the message(s) to the `pulserl_conn` process. When the\nconnection process receives the response it will `!` send it to the associated producer which in\nturn dequeue the associated caller and reply to it.\nThe producer provides synchronous and asynchronous send API.\n\n\nIn synchronous mode, the call will wait for the broker to acknowledge the message.\nIf the acknowledgment is not received and a `send_timeout` is specified, a `{error, send_timeout}`\nis sent to client on timed out.\n\nThe asynchronous mode provides two APIs. One returns a `reference()` that will be used to probe\nfor a response or error. The other allows one to pass a callback `fun/1` that will be invoked\ninternally by the producer process when there is a response or error.\n\n#### Starting a producer\nStart a producer with `pulserl:start_producer/1,2` by passing a topic name and list of options.\nIf `start_producer/1` is used, the producer will be started with a default options which can be provided\nas an environment variable in `sys.config`.\n\n```erlang\n[\n  {pulserl, [\n    {producer_opts, [{batch_enable, true}]}\n]}\n]\n```\n\nA sample start producer API code:\n```erlang\n  ProducerOpts = [\n    {producer, [\n      {name, \"my_producer_name\"}, %% Name of producer in the pulsar cluster\n       %% Metadata attached to the producer for easier identification\n        {properties, #{\"language\" =\u003e \"erlang\"}}, %% This can be proplist of as well\n         %% The initial sequence id for the producer\n        {initial_sequence_id, 0}, %% Default with 0\n    ]},\n     %% The time duration in milliseconds after which if the broker does not\n     %% acknowledge an error will be reported. The default is 30000\n    {send_timeout, 20000},\n     %% This behaviour is applied when a message without a key is to be published.\n     %% If the key is set, then the hash of the key will be used to choose\n     %% which partition the message will be published to.\n     %% Possible values: round_robin_routing, single_routing\n     %% and {Module, Function} which will be called with the key and\n     %% partition count to return the selected partition.\n    {routing_mode, round_robin_routing}, %% Default with round_robin_routing\n    {batch_enable, true} %% Default is true\n     %% The time duration (milliseconds) within which the messages sent will be batched.\n    {batch_max_delay_ms, 100}, %% Default is 10\n     %% Maximum number of messages that can be in a batch\n    {batch_max_messages, 100}, %% Default is 1000\n     %% Max size of the queue of requests waiting for acknowledgement.\n    {max_pending_requests, 50000}, %% Default is 100000\n  ],\n\n  {ok, Pid1} = pulserl:start_producer(\"topic-name\"), %% Will use default option values\n  {ok, Pid2} = pulserl:start_producer(\"persistent://public/default/topic-name\", ProducerOpts),\n\n```\n\n#### Producing messages\n\n```erlang\nMessage = \u003c\u003c\"message\"\u003e\u003e,\nTopic = topic_utils:parse(\"test-topic\"),\npulserl:sync_produce(Topic, Message).\n```\n\n### Consumer\n\nSimilar to producer `pulserl:start_consumer/1,2` starts a \"parent\" `gen_server` under the `pulserl_consumer_sup`\nwhich in turn `start_link` and manage `n` child consumers per partition, parent consumer serves as a facade to the internal consumers.\nThe parent monitor the child processes (internal partitioned consumers) for resilience.\n\n#### Starting a consumer\nStart a consumer with `pulserl:start_consumer/1,2` by passing a topic name and list of options.\nIf `start_consumer/1` is used, the consumer will be started with a default options which can be provided in `sys.config` using the `consumer_opts` environment variable.\n\n```erlang\n[\n  {pulserl, [\n    {consumer_opts, [{queue_size, 100}]}\n]}\n]\n```\n\nA sample start consumer API code:\n```erlang\n  ConsumerOpts = [\n    {queue_size, 100},  %% Default is 1000\n    {subscription_name, \"pulserl-app\"}, %% Default is \"pulserl\"\n    {subscription_type, 'Failover'} %% Default is 'Shared'\n    {acknowledgment_timeout, 1000}, %% Default is 0 (disabled)\n    {nack_message_redelivery_delay, 30000} %% Default is 60000\n    {dead_letter_topic_max_redeliver_count, 100} %% Default is 0 (disabled)\n  ],\n\n  {ok, Pid1} = pulserl:start_consumer(\"topic-name\"), %% Will use default option values\n  {ok, Pid2} = pulserl:start_consumer(\"persistent://public/default/topic-name\", ConsumerOpts),\n\n```\n\nOn the start, every consumer process automatically sends [subscription command](https://pulsar.apache.org/docs/en/2.6.0/develop-binary-protocol/#consumer) to the broker, with [shared](https://pulsar.apache.org/docs/en/2.6.0/concepts-messaging/#shared) subscription type as default.\nAfter subscribtion command consumers sends the [flow command](https://pulsar.apache.org/docs/en/2.6.0/develop-binary-protocol/#flow-control) with the value of the consumer's message queue(`not Erlang process's`) len. At every point, the number of messages in the queue is \u003c= `queue_size` and are readily available for upstream consumption. The consumer will resend a new flow permits every time the queue's size reaches the specified `queue_refill_threshold` (default to 50% of `queue_size`).\nTo receive a message `pulserl:consume/1` should be used in a loop, for example:\n\n```erlang\nreceive_message(PidOrTopic) -\u003e\n  case pulserl:consume(PidOrTopic) of\n    #consMessage{} = ConsumerMsg -\u003e\n      pulserl:ack(ConsumerMsg),\n      ConsumerMsg;\n    {error, _} = Error -\u003e\n      error(Error);\n    _ -\u003e\n      receive_message(Pid)\n  end.\n```\nwhere `PidOrTopic` is a pid of parent consumer process returned by `pulserl:start_consumer/1,2` or a topic name.\n\nEach received message should be [acknowledged](https://pulsar.apache.org/docs/en/concepts-messaging/#acknowledgement) with `pulserl:ack/1,2`.\n`pulsar:nack/1,2` could be used to ask the broker for [redelivering of the message](https://pulsar.apache.org/docs/en/concepts-messaging/#negative-acknowledgement), but it should be send before [acknowledgement timeout](https://pulsar.apache.org/docs/en/concepts-messaging/#acknowledgement-timeout) which is disabled by default.\nTo redeliver all the unacknowledged messages, one can use the `pulserl_consumer:redeliver_unack_messages/1` API\n\n## Contribute\n\nFor issues, comments, recommendation or feedback please [do it here](https://github.com/skulup/pulserl/issues).\n\nContributions are highly welcome.\n\n:thumbsup:\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonimesl%2Fpulserl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmonimesl%2Fpulserl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonimesl%2Fpulserl/lists"}