{"id":13481450,"url":"https://github.com/CleverCloud/pulsar4s","last_synced_at":"2025-03-27T12:30:51.793Z","repository":{"id":39984517,"uuid":"116223417","full_name":"CleverCloud/pulsar4s","owner":"CleverCloud","description":"Idiomatic, typesafe, and reactive Scala client for Apache Pulsar","archived":false,"fork":false,"pushed_at":"2024-08-27T15:06:39.000Z","size":841,"stargazers_count":227,"open_issues_count":49,"forks_count":45,"subscribers_count":12,"default_branch":"master","last_synced_at":"2024-10-29T23:34:22.293Z","etag":null,"topics":["akka-streams","async","cats","fs2","monix","pulsar","reactive","reactive-streams","scala","scalaz"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CleverCloud.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-01-04T06:29:12.000Z","updated_at":"2024-10-21T12:57:31.000Z","dependencies_parsed_at":"2024-11-12T17:00:30.256Z","dependency_job_id":"ade850ee-f3cb-430b-b009-cec7d23ab2c0","html_url":"https://github.com/CleverCloud/pulsar4s","commit_stats":{"total_commits":541,"total_committers":27,"mean_commits":"20.037037037037038","dds":0.5951940850277264,"last_synced_commit":"e2564847730e4e22a33c264442db6aaa8104cedf"},"previous_names":["sksamuel/pulsar4s"],"tags_count":34,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CleverCloud%2Fpulsar4s","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CleverCloud%2Fpulsar4s/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CleverCloud%2Fpulsar4s/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CleverCloud%2Fpulsar4s/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CleverCloud","download_url":"https://codeload.github.com/CleverCloud/pulsar4s/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245844836,"owners_count":20681786,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["akka-streams","async","cats","fs2","monix","pulsar","reactive","reactive-streams","scala","scalaz"],"created_at":"2024-07-31T17:00:51.936Z","updated_at":"2025-03-27T12:30:51.239Z","avatar_url":"https://github.com/CleverCloud.png","language":"Scala","funding_links":[],"categories":["Database","Table of Contents"],"sub_categories":["Database"],"readme":"# pulsar4s - Apache Pulsar Scala Client\n\n![build](https://github.com/CleverCloud/pulsar4s/workflows/build/badge.svg)\n[\u003cimg src=\"https://img.shields.io/maven-central/v/com.clever-cloud.pulsar4s/pulsar4s-core_3.svg?label=latest%20release%20for%203\"/\u003e](http://search.maven.org/#search%7Cga%7C1%7Ca%3A%22pulsar4s-core_3%22)\n[\u003cimg src=\"https://img.shields.io/maven-central/v/com.clever-cloud.pulsar4s/pulsar4s-core_2.13.svg?label=latest%20release%20for%202.13\"/\u003e](http://search.maven.org/#search%7Cga%7C1%7Ca%3A%22pulsar4s-core_2.13%22)\n[\u003cimg src=\"https://img.shields.io/maven-central/v/com.clever-cloud.pulsar4s/pulsar4s-core_2.12.svg?label=latest%20release%20for%202.12\"/\u003e](http://search.maven.org/#search%7Cga%7C1%7Ca%3A%22pulsar4s-core_2.12%22)\n[\u003cimg src=\"https://img.shields.io/nexus/s/https/oss.sonatype.org/com.clever-cloud.pulsar4s/pulsar4s-core_3.svg?label=latest%20snapshot\u0026style=plastic\"/\u003e](https://oss.sonatype.org/content/repositories/snapshots/com/sksamuel/pulsar4s/)\n[\u003cimg src=\"https://img.shields.io/nexus/s/https/oss.sonatype.org/com.clever-cloud.pulsar4s/pulsar4s-core_2.13.svg?label=latest%20snapshot\u0026style=plastic\"/\u003e](https://oss.sonatype.org/content/repositories/snapshots/com/sksamuel/pulsar4s/)\n\npulsar4s is a concise, idiomatic, reactive, type safe Scala client for [Apache Pulsar](https://pulsar.apache.org/).\nAs a simple wrapper over the Java client, we benefit from the reliability and performance of that client while providing better integration with the Scala ecosystem and idioms.\n\n* Supports different effects - [scala.concurrent.Future](https://docs.scala-lang.org/overviews/core/futures.html),\n[monix.eval.Task](https://monix.io/docs/2x/eval/task.html),\n[cats.effect.IO](https://typelevel.org/blog/2017/05/02/io-monad-for-cats.html),\n[scalaz.concurrent.Task](https://github.com/indyscala/scalaz-task-intro/blob/master/presentation.md)\n* Uses scala.concurrent.duration.Duration\n* Provides case classes rather than Java beans\n* [Akka Streams](https://github.com/sksamuel/pulsar4s#akka-streams) source and sink\n* [FS2](https://github.com/typelevel/fs2) Reader and Writer\n* Circe, SprayJson, PlayJson and Jackson implementations of Schema typeclass\n\n## Warning!!\n\n(This disclaimer was written on 2023-01-05.)\n\nStarting in version 2.9.0, we support scala 3. This means we had to perform some\n\"aggressive\" bumps on libs:\n\nLibs that were bumped for everyone:\n- play-json 2.10 (Currently in RC7)\n- cats-effect 3.3 (was 2.x)\n- ZIO 2.0 (was 1.x) \u0026 zio-cats-interop 23.0.0\n\nLibs that come in different versions across scala versions:\n- avro4s\n  - for Scala 3: 5.0+\n  - for Scala 2: 4.1+\n- scala-java8-compat\n  - for Scala ≥2.13: 1.0.2\n  - for Scala 2.12: 0.8.0 (Was already the case before scala 3)\n\nCheck carefully that bumping pulsar4s will not break, especially with **cats-effect**! \n\n## Using the client\n\nThe first step is to create a client attached to the pulsar cluster, providing the service url.\n\n```scala\nval client = PulsarClient(\"pulsar://localhost:6650\")\n```\n\nAlternatively, you can use an instance of `PulsarClientConfig` if you need to set further configuration\noptions such as authentication, tls, timeouts and so on.\n\n```scala\nval config = PulsarClientConfig(\"pulsar://localhost:6650\", ...)\nval client = PulsarClient(config)\n```\n\nThen we can create either a producer or a consumer from the client. We need an implicit schema in scope - more on that later.\n\nTo create a producer, we need the topic, and an instance of `ProducerConfig`.\nWe can set further options on the config object, such as max pending messages, router mode, producer name and so on.\n\n```scala\nimplicit val schema: Schema[String] = Schema.STRING\n\nval topic = Topic(\"persistent://sample/standalone/ns1/b\")\nval producerConfig = ProducerConfig(topic, ...)\nval producer = client.producer[String](producerConfig)\n```\n\nTo create a consumer, we need one or more topics to subscribe to, the subscription name, and an instance of `ConsumerConfig`.\nWe can set further options on the config object, such as subscription type, consumer name, queue size and so on.\n\n```scala\nimplicit val schema: Schema[String] = Schema.STRING\n\nval topic = Topic(\"persistent://sample/standalone/ns1/b\")\nval consumerConfig = ConsumerConfig(Seq(topic), Subscription(\"mysub\"), ...)\nval consumerFn = client.consumer[String](ConsumerConfig(, )\n```\n\nNote: Call `close()` on the client, producer, and consumer once you are finished. The client and producer also implement `AutoCloseable` and `Closeable`.\n\n### Schemas\n\nA message must be the correct type for the producer or consumer. When a producer or consumer is created,\nan implicit `Schema` typeclass must be available. In the earlier examples, you saw that we added an implicit schema for String using `implicit val schema: Schema[String] = Schema.STRING`.\n\nThere are built in schemas for bytes and strings, but other complex types required a custom schema.\nSome people prefer to write custom typeclasses manually for the types they need to support.\nOther people like to just have it done automagically. For those people, pulsar4s provides extensions\nfor the well known Scala Json libraries that can be used to generate messages where the body\nis a JSON representation of the class.\n\nAn example of creating a producer for a complex type using the circe json library to generate the schema:\n\n```scala\nimport io.circe.generic.auto._\nimport com.sksamuel.pulsar4s.circe._\n\nval topic = Topic(\"persistent://sample/standalone/ns1/b\")\nval producer = client.producer[Food](ProducerConfig(topic))\nproducer.send(Food(\"pizza\", \"ham and pineapple\"))\n```\n\nNote: The imports bring into scope a method that will generate an implicit schema when required.\n\nThe following extension modules can be used for automatic schemas\n\n| Library | Module | Import |\n|---------|------------------|--------|\n|[Circe](https://github.com/travisbrown/circe)|[pulsar4s-circe](http://search.maven.org/#search%7Cga%7C1%7Cpulsar4s-circe)|import io.circe.generic.auto._ \u003cbr/\u003eimport com.sksamuel.pulsar4s.circe._|\n|[Jackson](https://github.com/FasterXML/jackson-module-scala)|[pulsar4s-jackson](http://search.maven.org/#search%7Cga%7C1%7Cpulsar4s-jackson)|import com.sksamuel.pulsar4s.jackson._|\n|Json4s|[pulsar4s-json4s](http://search.maven.org/#search%7Cga%7C1%7Cpulsar4s-json4s)|import com.sksamuel.pulsar4s.json4s._|\n|Spray Json|[pulsar4s-spray-json](http://search.maven.org/#search%7Cga%7C1%7Cpulsar4s-spray-json)|import com.sksamuel.pulsar4s.sprayjson._|\n|Play Json|[pulsar4s-play-json](http://search.maven.org/#search%7Cga%7C1%7Cpulsar4s-play-json)|import com.sksamuel.pulsar4s.playjson._|\n\n\n### Producing\n\nThere are two ways to send a message - either with a plain value, or with an instance of `ProducerMessage`.\nIf you do not need to specify extra options on the message - such as key, event time, headers, etc - then you can just send\na plain value, and the client will wrap the value in a pulsar message. Alternatively, you can create an instance of `ProducerMessage`\nto specify extra options.\n\nEach method can be synchronous or asynchronous. The asynchronous methods return a `scala.concurrent.Future`.\nIf you are using another effect library, such as cats, scalaz or monix, then pulsar4s\nalso supports those effects. See the section on #effects.\n\nIf the send method is successful, you will receive the `MessageId` of the generated message. If an exception is generated, then in the synchronous methods, you will receive a `Failure` with the error. In the asynchronous\nmethods the exception will be surfaced as a failed Future.\n\nTo send a plain value, we just invoke `send` with the value:\n\n```scala\nproducer.send(\"wibble\")\n```\n\nOr to send a message, we first create an instance of `ProducerMessage`.\n\n```scala\nval message = DefaultProducerMessage(Some(\"mykey\"), \"wibble\", eventTime = Some(EventTime(System.currentTimeMillis)))\nproducer.send(message)\n```\n\n### Consuming\n\nTo receive a message, create a consumer and invoke either the `receive`, `receive(Duration)`, or the `receiveAsync` methods.\nThe first two are synchronous and return an instance of `ConsumerMessage`, blocking if necessary, and the latter is asynchronous, returning\na Future (or other effect) with the `ConsumerMessage` once ready.\n\n```scala\nval message: Message = consumer.receive\n```\n\nor\n\n```scala\nval message: Future[T] = consumer.receiveAsync\n```\n\nOnce a message has been consumed, it is important to acknowledge the message by using the message id with the ack methods.\n\n```scala\nconsumer.acknowledge(message.messageId)\n```\n\n\n## Akka Streams\n\nPulsar4s integrates with the outstanding [akka-streams](https://doc.akka.io/docs/akka/2.5.5/scala/stream/index.html) library - it provides both a source and a sink.\nTo use this, you need to add a dependency on the `pulsar4s-akka-streams` module.\n\n### Sources\n\nTo create a source all that is required is a function that will create a consumer on demand and the message id to seek.\nThe function must return a fresh consumer each time it is invoked.\nThe consumer is just a regular pulsar4s `Consumer` and can be created in the normal way, for example.\n\n```scala\nval topic = Topic(\"persistent://sample/standalone/ns1/b\")\nval consumerFn = () =\u003e client.consumer(ConsumerConfig(topic, subscription))\n```\n\nWe pass that function into the source method, providing the seek. Note the imports.\n\n```scala\nimport com.sksamuel.pulsar4s.akka.streams._\nval pulsarSource = source(consumerFn, Some(MessageId.earliest))\n```\n\nThe materialized value of the source is an instance of `Control` which provides a method called 'close' which can be used to stop consuming messages.\nOnce the akka streams source is completed (or fails) the consumer will be automatically closed.\n\n### Sinks\n\nTo create a sink, we need a producer function similar to the source's consumer function.\nAgain, the producer used is just a regular pulsar4s `Producer`.\nThe function must return a fresh producer each time it is invoked.\n\n```scala\nval topic = Topic(\"persistent://sample/standalone/ns1/b\")\nval producerFn = () =\u003e client.producer(ProducerConfig(topic))\n```\n\nWe pass that function into the sink method. Once again, take note of the imports.\n\n```scala\nimport com.sksamuel.pulsar4s.akka.streams._\nval pulsarSink = sink(producerFn)\n```\n\nA sink requires messages of type `ProducerMessage[T]` where T is the value type of the message. For example, if we were producing\nString messages, then we would map our upstream messages into instances of `ProducerMessage[String]` before passing them to the sink.\n\n```scala\nimport com.sksamuel.pulsar4s.akka.streams._\nSource.fromIterator(() =\u003e List(\"a\", \"b\", \"c\", \"d\").iterator)\n  .map(string =\u003e ProducerMessage(string))\n  .runWith(sink(producerFn))\n```\n\nA sink will run until the upstream source completes. In other words, to terminate the sink, the source must be cancelled or completed.\nOnce the sink completes the producer will be automatically closed.\n\nThe materialized value of the sink is a `Future[Done]` which will be completed once the upstream source has completed.\n\nThere is also an implementation of a 'multi-sink'.\nMulti-sink allows to produce to multiple topics in Pulsar, while using just 1 sink.\nMulti-sink expects, in addition to `ProducerMessage[T]`, a `Topic`, so the input format is `(Topic, ProducerMessage[T])`.\nAll producers in the sink are lazily-created, once a tuple with a new topic is received.\nThere is also a possibility to provide a collection of topics in the constructing function, to create those topics\nahead of time if the names are known. New topics read from the stream will also be created on-the-fly.\n\nExample usage of a multi-sink:\n\n```scala\nimport com.sksamuel.pulsar4s.akka.streams._\n\nval topic1 = Topic(\"persistent://sample/standalone/ns1/b\")\nval topic2 = Topic(\"persistent://sample/standalone/ns1/bb\")\nval producerFn = (topic: Topic) =\u003e client.producer(ProducerConfig(topic))\nval pulsarMultiSink = multiSink(producerFn)\n# or to create those topics ahead of time:\nval pulsarMultiSink2 = multiSink(producerFn, Set(topic1, topic2))\n```\n\n### Full Example\n\nHere is a full example of consuming from a topic for 10 seconds, publising the messages back into another topic.\nObviously this is a bit of a toy example but shows everything in one place.\n\n```scala\nimport com.sksamuel.pulsar4s.{ConsumerConfig, MessageId, ProducerConfig, PulsarClient, Subscription, Topic}\nimport org.apache.pulsar.client.api.Schema\n\nimplicit val system: ActorSystem = ActorSystem()\nimplicit val materializer: ActorMaterializer = ActorMaterializer()\nimplicit val schema: Schema[Array[Byte]] = Schema.BYTES\n\nval client = PulsarClient(\"pulsar://localhost:6650\")\n\nval intopic = Topic(\"persistent://sample/standalone/ns1/in\")\nval outtopic = Topic(\"persistent://sample/standalone/ns1/out\")\n\nval consumerFn = () =\u003e client.consumer(ConsumerConfig(Seq(intopic), Subscription(\"mysub\")))\nval producerFn = () =\u003e client.producer(ProducerConfig(outtopic))\n\nval control = source(consumerFn, Some(MessageId.earliest))\n                .map { consumerMessage =\u003e ProducerMessage(consumerMessage.data) }\n                .to(sink(producerFn)).run()\n\nThread.sleep(10000)\ncontrol.close()\n```\n\n## FS2 Support\n\nPulsar4s integrates with the [fs2](https://fs2.io/) library - it provides both a source and a sink.\nTo use this, you need to add a dependency on the `pulsar4s-{effect}` + `pulsar4s-fs2` module.\n\n### Example\n\n```scala\nimport com.sksamuel.pulsar4s._\nimport com.sksamuel.pulsar4s.cats.CatsAsyncHandler._\nimport com.sksamuel.pulsar4s.fs2.Streams\n\nimport org.apache.pulsar.client.api.Schema\n\nimplicit val schema: Schema[Array[Byte]] = Schema.BYTES\n\nval client = PulsarClient(\"pulsar://localhost:6650\")\n\nval intopic = Topic(\"persistent://sample/standalone/ns1/in\")\nval outtopic = Topic(\"persistent://sample/standalone/ns1/out\")\n\nStreams.batch[IO, Array[Byte]](client.consumerAsync[Array[Byte], IO](ConsumerConfig(\n  subscriptionName = Subscription(\"mysub\"),\n  topics = Seq(intopic),\n  subscriptionInitialPosition = Some(SubscriptionInitialPosition.Earliest)\n)))\n  .map(_.map(ProducerMessage(_.value)))\n  .through(Streams.committableSink(client.producerAsync[Array[Byte], IO](ProducerConfig(outtopic))))\n  .compile\n  .drain\n\n```\n\n## Example SBT Setup\n\n```scala\nval pulsar4sVersion = \"x.x.x\"\nlibraryDependencies ++= Seq(\n  \"com.clever-cloud.pulsar4s\" %% \"pulsar4s-core\" % pulsar4sVersion,\n\n  // for the akka-streams integration\n  \"com.clever-cloud.pulsar4s\" %% \"pulsar4s-akka-streams\" % pulsar4sVersion,\n\n  // if you want to use avro for schemas\n  \"com.clever-cloud.pulsar4s\" %% \"pulsar4s-avro\" % pulsar4sVersion,\n\n  // if you want to use circe for schemas\n  \"com.clever-cloud.pulsar4s\" %% \"pulsar4s-circe\" % pulsar4sVersion,\n\n  // if you want to use json4s for schemas\n  \"com.clever-cloud.pulsar4s\" %% \"pulsar4s-json4s\" % pulsar4sVersion,\n\n  // if you want to use jackson for schemas\n  \"com.clever-cloud.pulsar4s\" %% \"pulsar4s-jackson\" % pulsar4sVersion,\n\n  // if you want to use spray-json for schemas\n  \"com.clever-cloud.pulsar4s\" %% \"pulsar4s-spray-json\" % pulsar4sVersion,\n\n  // if you want to use play-json for schemas\n  \"com.clever-cloud.pulsar4s\" %% \"pulsar4s-play-json\" % pulsar4sVersion,\n\n  // if you want to use monix effects\n  \"com.clever-cloud.pulsar4s\" %% \"pulsar4s-monix\" % pulsar4sVersion,\n\n  // if you want to use scalaz effects\n  \"com.clever-cloud.pulsar4s\" %% \"pulsar4s-scalaz\" % pulsar4sVersion,\n\n  // if you want to use cats effects\n  \"com.clever-cloud.pulsar4s\" %% \"pulsar4s-cats-effect\" % pulsar4sVersion,\n\n  // if you want to use fs2\n  \"com.clever-cloud.pulsar4s\" %% \"pulsar4s-fs2\" % pulsar4sVersion,\n\n  // if you want to use zio\n  \"com.clever-cloud.pulsar4s\" %% \"pulsar4s-zio\" % pulsar4sVersion\n)\n```\n\n## Contributions\nContributions to pulsar4s are always welcome. Good ways to contribute include:\n\n* Raising bugs and feature requests\n* Improving the performance of pulsar4s\n* Adding to the documentation\n\n## License\n```\nThis software is licensed under the Apache 2 license, quoted below.\n\nCopyright 2017-2018 Stephen Samuel\n\nLicensed under the Apache License, Version 2.0 (the \"License\"); you may not\nuse this file except in compliance with the License. You may obtain a copy of\nthe License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT\nWARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the\nLicense for the specific language governing permissions and limitations under\nthe License.\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCleverCloud%2Fpulsar4s","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FCleverCloud%2Fpulsar4s","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCleverCloud%2Fpulsar4s/lists"}