{"id":22379138,"url":"https://github.com/sky-uk/kafka-topic-loader","last_synced_at":"2025-07-31T01:31:53.574Z","repository":{"id":37497032,"uuid":"151068221","full_name":"sky-uk/kafka-topic-loader","owner":"sky-uk","description":"Reads the contents of provided Kafka topics","archived":false,"fork":false,"pushed_at":"2023-09-12T18:19:48.000Z","size":356,"stargazers_count":4,"open_issues_count":16,"forks_count":1,"subscribers_count":62,"default_branch":"master","last_synced_at":"2023-09-12T19:19:14.220Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sky-uk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-10-01T10:00:51.000Z","updated_at":"2023-09-12T19:19:14.221Z","dependencies_parsed_at":"2023-02-16T09:15:29.253Z","dependency_job_id":null,"html_url":"https://github.com/sky-uk/kafka-topic-loader","commit_stats":null,"previous_names":[],"tags_count":28,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sky-uk%2Fkafka-topic-loader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sky-uk%2Fkafka-topic-loader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sky-uk%2Fkafka-topic-loader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sky-uk%2Fkafka-topic-loader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sky-uk","download_url":"https://codeload.github.com/sky-uk/kafka-topic-loader/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228204605,"owners_count":17884711,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-04T23:09:05.729Z","updated_at":"2025-07-31T01:31:53.563Z","avatar_url":"https://github.com/sky-uk.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kafka-topic-loader\n\n[![Build Status](https://app.travis-ci.com/sky-uk/kafka-topic-loader.svg?branch=master)](https://app.travis-ci.com/sky-uk/kafka-topic-loader)\n[![Maven Central](https://img.shields.io/maven-central/v/uk.sky/kafka-topic-loader_2.13?color=orange)](https://mvnrepository.com/artifact/uk.sky/kafka-topic-loader)\n[![Sonatype Nexus (Snapshots)](https://img.shields.io/nexus/s/uk.sky/kafka-topic-loader_2.13?label=snapshot\u0026server=https%3A%2F%2Fs01.oss.sonatype.org)](https://s01.oss.sonatype.org/content/repositories/snapshots/uk/sky/kafka-topic-loader_2.13/)\n\nReads the contents of provided Kafka topics, either the topics in their entirety or up until a consumer groups last committed Offset depending on which `LoadTopicStrategy` you provide.\n\nAs of version `1.3.0`, data can be loaded either from complete topics using `load` or `loadAndRun`.\n\nSince version `1.4.0` the library is cross compiled for scala versions `2.12` and `2.13`.\n\nSince version `1.6.0` the library is cross compiled for scala versions `2.12`, `2.13` and `3`.\n\nSince version `2.0.0` the library is no longer cross compiles for version `2.12`, and the package has been renamed from `com.sky` to `uk.sky`.\n\nAdd the following to your `build.sbt`:\n\n```scala\nlibraryDependencies += \"uk.sky\" %% \"kafka-topic-loader\" % \"\u003cversion\u003e\"\n```\n\n```scala\nimport uk.sky.kafka.topicloader.{LoadAll, TopicLoader}\nimport org.apache.kafka.common.serialization.Deserializer}\n\nimplicit val as: ActorSystem = ActorSystem()\nimplicit val stringDeserializer: Deserializer[String] = new StringDeserializer\n\nval stream = TopicLoader.load[String, String](NonEmptyList.one(\"topic-to-load\"), LoadAll)\n      .mapAsync(1)(_ =\u003e ??? /* store records in pekko.Actor for example */)\n      .runWith(Sink.ignore)\n```\n\n`loadAndRun` will load the topics, complete the `Future[Done]` from the materialised value and then carry on\nrunning, emitting any new records that appear on the topics. An example use-case for this is a REST API that holds the\ncontents of a Kafka topic in memory. This kind of application doesn't need to commit offsets and can use the `Future[Done]` to determine readiness.\n\n```scala\nobject Main extends App {\n\n  implicit val system = ActorSystem()\n  implicit val mat    = ActorMaterializer()\n\n  import system.dispatcher\n\n  implicit val keyDeserializer: Deserializer[String]        = new StringDeserializer\n  implicit val valueDeserializer: Deserializer[Array[Byte]] = new ByteArrayDeserializer\n\n  val state = new SimplifiedState\n\n  val (initialLoadingFuture, controlF): (Future[Done], Future[Consumer.Control]) =\n    TopicLoader\n      .loadAndRun[String, Array[Byte]](NonEmptyList.one(\"topic-to-load\"))\n      .to(Sink.foreach(record =\u003e state.store.put(record.key, record.value)))\n      .run()\n\n  initialLoadingFuture.foreach(_ =\u003e state.isAppReady.set(true))\n}\n\nclass SimplifiedState {\n\n  /**\n    * API requests may query this state\n    */\n  val store = new ConcurrentHashMap[String, Array[Byte]]()\n\n  /**\n    * A readiness endpoint could be created that queries this\n    */\n  val isAppReady = new AtomicBoolean()\n}\n```\n\n## Configuration\n\n### Topic loader\n\nThe config in [`reference.conf`](src/main/resources/reference.conf) can be overridden by providing your own `application.conf`.\n\nBy default, Pekko's `ConsumerConfig` will inherit the consumer `client.id` from the application kafka-topic-loader is running from. To separate the client id of your application and the kafka-topic-loader, provide it in your `application.conf`:\n\n```hocon\ntopic-loader {\n  client-id = \"custom-client-id\"\n}\n```\n\n### Pekko-kafka\n\nYou should configure the `pekko.kafka.consumer.kafka-clients.group.id` to match that of your application, e.g.:\n\n```hocon\npekko.kafka {\n  consumer.kafka-clients {\n    bootstrap.servers = ${?KAFKA_BROKERS}\n    group.id = assembler-consumer-group\n  }\n  producer.kafka-clients {\n    bootstrap.servers = ${?KAFKA_BROKERS}\n  }\n}\n```\n\n## Source per partition\n\nThis is deprecated in favour of a new API for partitioned loading which is coming soon.\n\nData can also be loaded from specific partitions using `fromPartitions`. By loading from specific partitions the topic\nloader can be used by multiple application instances with separate streams per set of partitions (see [Pekko Connectors kafka](https://pekko.apache.org/docs/pekko-connectors-kafka/current/consumer.html#source-per-partition) and below).\n\n```scala\nimplicit val system = ActorSystem()\n\nval consumerSettings: ConsumerSettings[String, Long]              = ???\nval doBusinessLogic: ConsumerRecord[String, Long] =\u003e Future[Unit] = ???\n\nval stream: Source[ConsumerMessage.CommittableMessage[String, Long], Consumer.Control] =\n  Consumer\n    .committablePartitionedSource(consumerSettings, Subscriptions.topics(\"topic-to-load\"))\n    .flatMapConcat {\n      case (topicPartition, source) =\u003e\n        TopicLoader\n          .fromPartitions(LoadAll, NonEmptyList.one(topicPartition), doBusinessLogic, new LongDeserializer())\n          .flatMapConcat(_ =\u003e source)\n    }\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsky-uk%2Fkafka-topic-loader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsky-uk%2Fkafka-topic-loader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsky-uk%2Fkafka-topic-loader/lists"}