{"id":22379116,"url":"https://github.com/sky-uk/fs2-kafka-topic-loader","last_synced_at":"2026-02-26T20:41:51.959Z","repository":{"id":194039020,"uuid":"638491985","full_name":"sky-uk/fs2-kafka-topic-loader","owner":"sky-uk","description":"Reads the contents of provided Kafka topics","archived":false,"fork":false,"pushed_at":"2024-09-09T18:01:29.000Z","size":165,"stargazers_count":2,"open_issues_count":13,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-09-09T21:58:05.235Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sky-uk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-09T13:26:04.000Z","updated_at":"2024-09-09T17:50:06.000Z","dependencies_parsed_at":"2023-10-03T19:40:55.794Z","dependency_job_id":"e3dba232-b3bd-4bd9-a5a1-860e045944bc","html_url":"https://github.com/sky-uk/fs2-kafka-topic-loader","commit_stats":null,"previous_names":["sky-uk/fs2-kafka-topic-loader"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sky-uk%2Ffs2-kafka-topic-loader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sky-uk%2Ffs2-kafka-topic-loader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sky-uk%2Ffs2-kafka-topic-loader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sky-uk%2Ffs2-kafka-topic-loader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sky-uk","download_url":"https://codeload.github.com/sky-uk/fs2-kafka-topic-loader/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228204606,"owners_count":17884711,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-04T23:09:00.697Z","updated_at":"2026-02-26T20:41:46.918Z","avatar_url":"https://github.com/sky-uk.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://app.travis-ci.com/sky-uk/fs2-kafka-topic-loader.svg?branch=main)](https://app.travis-ci.com/sky-uk/fs2-kafka-topic-loader)\n[![Maven Central](https://img.shields.io/maven-central/v/uk.sky/fs2-kafka-topic-loader_3)](https://mvnrepository.com/artifact/uk.sky/fs2-kafka-topic-loader)\n[![Sonatype Nexus (Snapshots)](https://img.shields.io/nexus/s/uk.sky/fs2-kafka-topic-loader_3?label=snapshot\u0026server=https%3A%2F%2Fs01.oss.sonatype.org)](https://s01.oss.sonatype.org/content/repositories/snapshots/uk/sky/fs2-kafka-topic-loader_3/)\n\n# fs2-kafka-topic-loader\n\nReads the contents of provided Kafka topics.\n\nThis library is aimed for usage in applications that want a deterministic stream of Kafka messages that completes once\nthe last message (determined above) has been read. This is useful if an application shouldn't respond to new events\nbefore it has processed all previously seen messages, or if Kafka is being used as a data store and the entire contents\nof a topic needs to be reloaded on an application restart.\n\n## Usage\n\n### `Load`\n\nDetermined by the `LoadTopicStrategy`, this stream completes once all messages have been read.\n\nLoad strategies:\n\n- `LoadAll` - reads the topics in their entirety\n- `LoadCommitted` - reads up to the configured consumer-group's last committed Offset\n\n### `LoadAndRun`\n\nRead up to the latest offset, fire a callback and continue streaming messages.\n\n## Example\n\nAdd the following to your `build.sbt`:\n\n```scala\nlibraryDependencies += \"uk.sky\" %% \"fs2-kafka-topic-loader\" % \"\u003cversion\u003e\"\n```\n\n### Load\n\n```scala\nimport cats.data.NonEmptyList\nimport cats.effect.{IO, IOApp}\nimport fs2.kafka.ConsumerSettings\nimport org.typelevel.log4cats.LoggerFactory\nimport uk.sky.fs2.kafka.topicloader.{LoadAll, TopicLoader}\n\nobject Main extends IOApp.Simple {\n  val consumerSettings: ConsumerSettings[IO, String, String] = ???\n\n  given LoggerFactory[IO] = ???\n\n  override def run: IO[Unit] =\n    TopicLoader.load(NonEmptyList.one(\"topicToLoad\"), LoadAll, consumerSettings).evalTap(IO.println).compile.drain\n}\n```\n\nSee [`LoadExample.scala`](./it/src/main/scala/load/LoadExample.scala) for a more detailed example.\n\n### LoadAndRun\n\n```scala\nimport cats.data.NonEmptyList\nimport cats.effect.Resource.ExitCase\nimport cats.effect.{IO, IOApp, Ref}\nimport fs2.kafka.ConsumerSettings\nimport org.typelevel.log4cats.LoggerFactory\nimport uk.sky.fs2.kafka.topicloader.{LoadAll, TopicLoader}\n\nobject Main extends IOApp.Simple {\n  val consumerSettings: ConsumerSettings[IO, String, String] = ???\n\n  given LoggerFactory[IO] = ???\n\n  val healthCheck: IO[Ref[IO, Boolean]] = Ref.of(false)\n\n  val logger = LoggerFactory[IO].getLogger\n\n  override def run: IO[Unit] =\n    for {\n      healthCheck \u003c- healthCheck\n      _           \u003c- TopicLoader\n                       .loadAndRun(NonEmptyList.one(\"topicToLoad\"), consumerSettings) {\n                         case ExitCase.Succeeded  =\u003e healthCheck.set(true)\n                         case ExitCase.Errored(e) =\u003e logger.error(e)(s\"Something went wrong: $e\")\n                         case ExitCase.Canceled   =\u003e logger.warn(\"Stream was cancelled before loading\")\n                       }\n                       .compile\n                       .drain\n    } yield ()\n}\n```\n\n## Configuration\n\nConfiguration from the Topic Loader is done via the `ConsumerSettings`. The group id of the Topic Loader should match\nthe group id of your application.\n\n## Contributing\n\nContributions are welcomed! For contributions see [here](./CONTRIBUTING.md) for more information.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsky-uk%2Ffs2-kafka-topic-loader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsky-uk%2Ffs2-kafka-topic-loader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsky-uk%2Ffs2-kafka-topic-loader/lists"}