https://github.com/evolution-gaming/kafka-flow
library for reliable processing records received from kafka
https://github.com/evolution-gaming/kafka-flow
cats cats-effect kafka kafka-streams scala
Last synced: 3 months ago
JSON representation
library for reliable processing records received from kafka
- Host: GitHub
- URL: https://github.com/evolution-gaming/kafka-flow
- Owner: evolution-gaming
- License: mit
- Created: 2019-08-28T19:36:59.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2025-04-03T10:20:30.000Z (3 months ago)
- Last Synced: 2025-04-23T00:15:45.041Z (3 months ago)
- Topics: cats, cats-effect, kafka, kafka-streams, scala
- Language: Scala
- Homepage:
- Size: 1.77 MB
- Stars: 21
- Watchers: 12
- Forks: 11
- Open Issues: 21
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# kafka-flow
[](https://github.com/evolution-gaming/kafka-flow/actions?query=workflow%3ACI)
[](https://coveralls.io/github/evolution-gaming/kafka-flow?branch=master)
[](https://app.codacy.com/gh/evolution-gaming/kafka-flow/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade)
[](https://evolution.jfrog.io/artifactory/api/search/latestVersion?g=com.evolutiongaming&a=kafka-flow_2.13&repos=public)
[](https://opensource.org/licenses/MIT)## Microsite
https://evolution-gaming.github.io/kafka-flow
## Scala 3 compatibility
Starting from version `6.1.0` all of the modules are cross-compiled to Scala 3 except `kafka-flow-kafka-journal` which doesn't support Scala 3 yet.## Setup
```scala
addSbtPlugin("com.evolution" % "sbt-artifactory-plugin" % "0.0.2")lazy val version = "" // see the latest one in the badge above or in Releases page
libraryDependencies ++= Seq(
"com.evolutiongaming" %% "kafka-flow" % version,
// if you want to use Cassandra for storing persistent state
"com.evolutiongaming" %% "kafka-flow-persistence-cassandra" % version,
// if you want to use Kafka compact topic for storing persistent state
"com.evolutiongaming" %% "kafka-flow-persistence-kafka" % version,
// if you want to use predefined metrics
"com.evolutiongaming" %% "kafka-flow-metrics" % version,
// if you want to use kafka-journal integration
"com.evolutiongaming" %% "kafka-flow-kafka-journal" % version,
)
```## Release process
The release process is based on Git tags and makes use of [evolution-gaming/scala-github-actions](https://github.com/evolution-gaming/scala-github-actions) which uses [sbt-dynver](https://github.com/sbt/sbt-dynver) to automatically obtain the version from the latest Git tag. The flow is defined in `.github/workflows/release.yml`.
A typical release process is as follows:
1. Create and push a new Git tag. The version should be in the format `vX.Y.Z` (example: `v4.1.0`). Example: `git tag v4.1.0 && git push origin v4.1.0`
2. On success, a new GitHub release is automatically created with a calculated diff and auto-generated release notes. You can see it on `Releases` page, change the description if needed
3. On failure, the tag is deleted from the remote repository. Please note that your local tag isn't deleted, so if the failure is recoverable then you can delete the local tag and try again (an example of *unrecoverable* failure is successfully publishing only a few of the artifacts to Artifactory which means a new attempt would fail since Artifactory doesn't allow overwriting its contents)