{"id":13846763,"url":"https://github.com/jeffgrunewald/stargate","last_synced_at":"2025-07-11T01:33:18.587Z","repository":{"id":39365738,"uuid":"217272064","full_name":"jeffgrunewald/stargate","owner":"jeffgrunewald","description":"An Apache Pulsar client written in Elixir","archived":false,"fork":false,"pushed_at":"2022-10-02T18:21:26.000Z","size":85,"stargazers_count":42,"open_issues_count":4,"forks_count":12,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-10-19T06:08:52.711Z","etag":null,"topics":["data-processing","elixir","pulsar-client"],"latest_commit_sha":null,"homepage":null,"language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jeffgrunewald.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":"jeffgrunewald"}},"created_at":"2019-10-24T10:23:53.000Z","updated_at":"2024-01-19T04:17:34.000Z","dependencies_parsed_at":"2022-07-04T07:21:18.169Z","dependency_job_id":null,"html_url":"https://github.com/jeffgrunewald/stargate","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeffgrunewald%2Fstargate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeffgrunewald%2Fstargate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeffgrunewald%2Fstargate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeffgrunewald%2Fstargate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jeffgrunewald","download_url":"https://codeload.github.com/jeffgrunewald/stargate/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225484528,"owners_count":17481550,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-processing","elixir","pulsar-client"],"created_at":"2024-08-04T18:00:46.820Z","updated_at":"2024-11-21T03:50:44.218Z","avatar_url":"https://github.com/jeffgrunewald.png","language":"Elixir","funding_links":["https://github.com/sponsors/jeffgrunewald"],"categories":["Clients"],"sub_categories":[],"readme":"![](https://github.com/jeffgrunewald/stargate/workflows/CI/badge.svg)\n\n# Stargate\n\nAn Apache Pulsar client written in Elixir using the Pulsar websocket API.\n\nStargate allows you to create producer and consumer connections to a Pulsar\ncluster and send messages to or receive messages from the cluster. Receivers\nof messages from Pulsar can be a Reader connection, receiving messages from a\ntopic based on an initial message in the log defined by the user, or via a Consumer\nconnection, receiving all messages on the topic after initial connection.\n\nPer Pulsar's documentation, all messages received from a topic are acknowledged by the websocket\nAPI, but the Reader acknowledgment is solely to signal to the socket the reader is\nready for additional messages, while Consumer acknowledgement will additionally signal\nto the socket the consumer is ready for the message to be deleted from its subscription.\n\nStargate receivers must supply a message handler module to tell Stargate how to handle\nmessages received from the socket.\n\nProducers must supply connection settings for a persistent connection or may\nsupply a single URL signifying the desired persistence, tenant, namespace, and topic\nto produce to when ad hoc produce is desired.\n\n## Installation\n\n[Available in Hex](https://hex.pm/docs/publish), the package can be installed\nby adding `stargate` to your list of dependencies in `mix.exs` via the Github strategy:\n\n```elixir\ndef deps do\n  [\n    {:stargate, \"~\u003e 0.1.0\"}\n  ]\nend\n```\n\nThe docs can be found at [https://hexdocs.pm/stargate](https://hexdocs.pm/stargate).\n\n## Usage\n### Locating Processes\nUnder the hood, Stargate uses the Registry module to create a via-tuple name for each process within a Stargate-\nmanaged supervision tree. The intended API is to create a top-level `Stargate.Supervisor` and attach option producers\nand receivers under it as needed. When you do this, Stargate creates the top-level supervisor and prepends `:sg_sup_#{name}`\nto the name of the process (defaults to `:default`) as well as a Registry named `:sg_reg_#{name}` (defaults to `:default`).\nAll processes spun up under this pair receive a via-tuple name identified by `{via, Registry, {\u003cregistry_name\u003e, {\u003ccomponent\u003e,\n\u003cpersistence\u003e, \u003ctenant\u003e, \u003cnamespace\u003e, \u003ctopic\u003e}}}` where component is the type of process (`:producer`, `:consumer`, etc),\npersistence is either \"persistent\" or \"non-persistent\", and the remaining values are all strings representing the values supplied\nwhen creating the topic in Pulsar.\n\nTo assist in finding the correct process to send messages to, Stargate provides the `Stargate.registry_key/4` function as a\nhelper to calculate this via-tuple easily. When supplying the `Stargate.registry_key(\u003ctenant\u003e, \u003cnamespace\u003e, \u003ctopic\u003e)`, Stargate\nassumes you're trying to reach the Producer process as this is the process most likely to be called directly (in conjunction within\n`Stargate.produce/2,3`), assumes a persistent topic (because Pulsar assumes topics are persistent by default) and assumes the\nSupervisor and Registry name suffix to be `:default`. If necessary, you can supply a Keyword List of options to customize the\nvia-tuple returned by `Stargate.registry_key/4` with a combination of the arguments `:name` or `:registry`, `:component`, `:persistence`.\n\n```elixir\niex\u003e Stargate.registry_key(\"foo\", \"bar\", \"baz\")\n{:via, Registry, {:sg_reg_default, {:producer, \"persistent\", \"foo\", \"bar\", \"baz\"}}}\n\niex\u003e Stargate.registry(\"foo\", \"bar\", \"baz\", name: :custom, persistence: \"non-persistent\", component: :consumer)\n{:via, Registry, {:sg_reg_custom, {:consumer, \"non-persistent\", \"foo\", \"bar\", \"baz\"}}}\n```\n\n### Produce\nProducing to Pulsar via Stargate is as simple as passing an Erlang term to the produce function. Stargate\ntakes care of encoding the message payload with the necessary fields and format required by Pulsar with\noptions to specify certain fields as fits the users' specific needs (see below).\n\nTo ad hoc produce to Pulsar via Stargate, call `Stargate.produce(url, [message])`\nwhere `url` is something like ws://cluster-url:8080/ws/v2/producer/:persistence/:tenant/:namespace/:topic\"\nand `[message]` is a single message or a list of messages which can be one of:\n- a string-keyed map containing the \"payload\" key and optionally the \"context\" key if a specific\nmessage context is desired (Stargate uses the \"context\" for tracking receipt of produced messages by the cluster).\n- A two-element tuple containing a key as the first element and a payload as the second.\n- A message payload only\nStargate will clean up the producer processes once the produce has completed.\n\nFor persistent producer connections to the cluster, you may also start a supervised tree of processes\nincluding the socket producer itself and an acknowledger, the job of which is to ensure receipt of messages\nby the cluster.\n\nTo start a supervised producer, call something like the following within your application:\n```elixir\nopts = [\n  name: :my_producer,               optional \\\\ default :default\n  host: [\"example.com\": 8080],\n  protocol: \"ws\",                     optional \\\\ default ws\n  producer: [\n    persistence: \"persistent\",        optional \\\\ default persistent\n    tenant: \"public\",\n    namespace: \"default\",\n    topic: \"foo\",\n    query_params: %{                  all query params are optional\n      name: \"summer\",\n      send_timeout: 30_000,             \\\\ default 30_000\n      batch_enabled: true,              \\\\ default false\n      batch_max_msg: 1_000,             \\\\ default 1_000\n      max_pending_msg: 1_000,           \\\\ default 1_000\n      batch_max_delay: 25,              \\\\ default 10\n      routing_mode: :round_robin,       \\\\ (deprecated by Pulsar) options :round_robin | :single\n      compression_type: :lz4,           \\\\ default uncompressed, options :lz4 | :zlib\n      producer_name: \"myapp-producer\",\n      initial_seq_id: 100,\n      hashing_scheme: :murmur3          \\\\ options :java_string | :murmur3\n    }\n  ]\n]\nStargate.Supervisor.start_link(opts)\n```\n\nBy default, the `Stargate.produce/2` will block until it receives\nacknowledgement the message has successfully been produced to the cluster. To ackowledge produce\noperations in a non-blocking manner, you may also use `Stargate.produce/3` and pass an MFA tuple\nas the third argument instructing the acknowledger which module, function, and arguments to execute\nupon receipt of acknowledgement from Pulsar that the produce succeeded.\n\n### Receive (Consume/Read)\nTo consume messages from Pulsar via Stargate, your application needs to create a consumer or reader\nprocess by calling something like the following:\n```elixir\n  opts = [\n    name: :my_reader,               optional \\\\ default :default\n    host: [\"example.com\": 8080],\n    protocol: \"ws\",                  optional \\\\ default ws\n    reader: [\n      persistence: \"persistent\",     optional \\\\ default persistent\n      tenant: \"public\",\n      namespace: \"default\",\n      topic: \"foo\",\n      processors: 3,                 optional \\\\ default 1\n      handler: MyApp.Reader.Handler,\n      handler_init_args: []          optional \\\\ default []\n      query_params: %{               all query params are optional\n        queue_size: 1_000,             \\\\ default 1_000\n        name: \"morty\",\n        starting_message: :latest      \\\\ default :latest, options :latest | :earliest | \"some-message-id\"\n      }\n    ]\n  ]\n\n  opts = [\n    name: :my_consumer,                optional \\\\ default :default\n    host: [\"example.com\": 8080],\n    protocol: \"ws\",                   optional \\\\ default ws\n    consumer: [\n      persistence: \"persistent\",      optional \\\\ default persistent\n      tenant: \"public\",\n      namespace: \"default\",\n      topic: \"foo\",\n      subscription: \"my-app\",\n      processors: 3,                  optional \\\\ default 1\n      handler: MyApp.Reader.Handler,\n      handler_init_args: []           optional \\\\ default []\n      query_params: %{                all query params are optional\n        subscription_type: :shared,     \\\\ default :exclusive\n        ack_timeout: 100,               \\\\ default 0\n        queue_size: 1_000,              \\\\ default 1_000\n        name: \"rick\",\n        priority: 10,\n        max_redeliver_count: 10,        \\\\ default 0\n        dead_letter_topic: \"ricks-dlq\", \\\\ default \"{topic}-{subscription}-DLQ\"\n        pull_mode: true                 \\\\ default false\n      }\n    ]\n  ]\n\n  Stargate.Supervisor.start_link(opts)\n```\n\nStargate's receivers are implemented using GenStage to allow for the message processing (the processes that actually\ncall your handler function on messages received) to be parallelized as well as to backpressure into the cluster\nwhen setting your consumer connection to `pull_mode: true`.\n\n### Producing AND Receiving\nStarting producers and consumer can but _need not_ require different supervision trees. If your application is a link\nin a chain that will both receive and produce to the same cluster, you can start a single supervisor with keys for\nboth the receiver and producer configs in the options passed to the `Stargate.Supervisor.start_link/1` function.\n\nAs an example, the configuration below would start a Stargate Supervisor that will manage processes that both monitor an\nhypothetical company's internal R\u0026D topic publishing application features ready for release and then turn around and\nproduce messages about those feature messages to the application marketing team's topic for public consumption.\n\n```elixir\n  options = [\n    name: :pulsar_app,\n    host: [{:\"example.com\", 8080}],\n    producer: [\n      persistence: \"non-persistent\",\n      tenant: \"marketing\",\n      namespace: \"public\",\n      topic: \"new-stuff\"\n    ]\n    consumer: [\n      tenant: \"internal\",\n      namespace: \"research\",\n      topic: \"ready-to-release\",\n      subscription: \"rss-feed\",\n      handler: Publicizer.MessageHandler\n    ]\n  ]\n\n  Stargate.Supervisor.start_link(options)\n```\n\n### Security\nStargate supports client security in two ways: by allowing TLS encryption of the connection between the client\nand the Pulsar cluster, and by JWT token authentication of the Websocket connection.\n\nIn order to allow TLS encryption of the websocket connection, the URL to the cluster must be over the secure\nport (Pulsar defaults to 8443) and using the `wss://` prefix instead of the plaintext `ws://`.\n\nTo pass custom certificates, to the client connection, include the following in the init options passed to the\nproducer or receiver connection:\n\n```elixir\n  options = [\n    ...\n    ssl_options: [\n      cacertfile: \"/certificates/cacert.pem\",\n      certfile: \"/certificates/cert.pem\",\n      keyfile: \"/certificates/key.pem\"\n    ],\n    ...\n  ]\n```\n\nIn order to pass a JWT token for authentication, include the following in your connection options for either\nthe producer or receiver process when starting the process.\n\n```elixir\n  options = [\n    ...\n    auth_token: \"some-jwt-token-string\",\n    ...\n  ]\n```\n\n### Receiving Messages\nThe message handler module passed to a receiver is expected to implement the `Stargate.Receiver.MessageHandler`\nbehaviour which includes an optional `init/1` if your message handler is expected to track state across\nmessages received from the topic and `handle_message/1`, `handle_message/2` where `handle_message/1` receives\nonly a message while `handle_message/2` receives a message and the state being tracked across all messages handled.\n\nIncluding `use Stargate.Receiver.MessageHandler` in your handler module will created default implementations of\nthese functions that can be overridden as needed. `handle_message/1` is expected to return either `:ack` or `:continue`\nwhile `handle_message/2` is expected to return `{:ack, state}` or `{:continue, state}`\n\nMessages received by Stargate are automatically converted to a `%Stargate.Message{}` struct by the time they reach\nthe message handler module. This struct contains all the information about where the message came from (topic, namespace,\ntenant, and persistence) as well as the message identifier assigned to it by the Pulsar cluster, the key if one was\nsupplied, any properties (key/value pairs) attached to the message by the producer, the timestamp the message was published\nby the cluster as an Elixir DateTime struct and of course the message payload decoded from the Base64 encoding received from\nPulsar.\n\n## Limitations\n\n### Cumulative acknowledgement\nAt present, Apache Pulsar does _not_ support cumulative acknowledgement of messages via\nthe websocket API. When this is supported (PR forthcoming with the upstream project), Stargate\nwill be updated to include this as an optional ack strategy for faster processing of large groups\nof messages.\n\n### Custom topic configuration and administrative functionality\nAt the time of initial release, Stargate only provides functionality for producing to and\nreceiving (reading or consuming) from Pulsar topics. Pulsar allows for automatically creating\nsingle-partition topics within existing tenants and namespaces.\n\nWhile Pulsar provides a RESTful management API, Stargate does not currently provide any\nfunctions for interacting with this API. This _is_ a planned expansion of Stargate functionality\nwith the intent that management of tenants, namespaces, and topics can all be accomplished\nnatively from Stargate modules.\n\nStay tuned!\n\n## Additional information\n### Testing\nStargate uses the [divo](https://github.com/smartcitiesdata/divo) and the plugin\n[divo_pulsar](https://github.com/jeffgrunewald/divo_pulsar) for integration testing,\nusing Docker to stand up a single-node Pulsar \"cluster\" in standalone mode for testing.\nSee the documentation on the divo and divo_pulsar hex packages for further details.\n\n### Pulsar\nFor additional information on Pulsar, including configuring and running a cluster,\nsee the project's [official documentation](https://pulsar.apache.org).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjeffgrunewald%2Fstargate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjeffgrunewald%2Fstargate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjeffgrunewald%2Fstargate/lists"}