Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/Banno/kafka4s
Functional programming with Kafka and Scala
https://github.com/Banno/kafka4s
cats cats-effect fp fs2 functional-programming kafka scala stream-processing
Last synced: 2 months ago
JSON representation
Functional programming with Kafka and Scala
- Host: GitHub
- URL: https://github.com/Banno/kafka4s
- Owner: Banno
- License: apache-2.0
- Created: 2019-06-21T15:30:50.000Z (over 5 years ago)
- Default Branch: main
- Last Pushed: 2024-10-29T19:49:04.000Z (2 months ago)
- Last Synced: 2024-10-29T21:42:48.828Z (2 months ago)
- Topics: cats, cats-effect, fp, fs2, functional-programming, kafka, scala, stream-processing
- Language: Scala
- Homepage: https://banno.github.io/kafka4s
- Size: 3.15 MB
- Stars: 92
- Watchers: 11
- Forks: 18
- Open Issues: 11
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Codeowners: CODEOWNERS
Awesome Lists containing this project
- awesome-ccamel - Banno/kafka4s - Functional programming with Kafka and Scala (Scala)
README
# kafka4s - Functional programming with Kafka and Scala
[](https://maven-badges.herokuapp.com/maven-central/com.banno/kafka4s_2.13)
[](https://www.javadoc.io/doc/com.banno/kafka4s_2.13/latest/com/banno/kafka/index.html)
[](http://www.apache.org/licenses/LICENSE-2.0.txt)
kafka4s provides pure, referentially transparent functions for working with Kafka, and integrates with FP libraries such as [cats-effect](https://typelevel.org/cats-effect) and [fs2](https://fs2.io).
## [Head on over to the microsite](https://banno.github.io/kafka4s)
## Quick Start
To use `kafka4s` in an existing SBT project with Scala 2.13 or a later version, add the following dependencies to your
`build.sbt` depending on your needs:
```scala
libraryDependencies ++= Seq(
"com.banno" %% "kafka4s" % ""
)
```
Note: If your project uses fs2 1.x, you'll want releases from the 2.x series. For fs2 2.x projects, you'll want 3.x series releases.
Sending records to Kafka is an effect. If we wanted to periodically write random integers to a Kafka topic, we could do:
```scala
Stream
.resource(ProducerApi.resource[F, Int, Int](BootstrapServers(kafkaBootstrapServers)))
.flatMap { producer =>
Stream
.awakeDelay[F](1.second)
.evalMap { _ =>
Sync[F].delay(Random.nextInt()).flatMap { i =>
producer.sendAndForget(new ProducerRecord(topic.name, i, i))
}
}
}
```
Polling Kafka for records is also an effect, and we can obtain a stream of records from a topic. We can print the even random integers from the above topic using:
```scala
Stream.resource(
ConsumerApi
.resource[F, Int, Int](
BootstrapServers(kafkaBootstrapServers),
GroupId("example3"),
AutoOffsetReset.earliest,
EnableAutoCommit(true)
)
)
.evalTap(_.subscribe(topic.name))
.flatMap(
_.recordStream(1.second)
.map(_.value)
.filter(_ % 2 == 0)
.evalMap(i => Sync[F].delay(println(i)))
)
```
## Learning more
To learn more about kafka4s, start with our [Getting Started Guide](https://banno.github.io/kafka4s/docs/), play with some [example apps](https://github.com/Banno/kafka4s/tree/master/examples/src/main/scala), and check out the [kafka4s Scaladoc](https://www.javadoc.io/doc/com.banno/kafka4s_2.13) for more info.
## Running the examples
To run the examples, setup the following:
- Pull down the docker image in the project directory:
```
docker-compose up -d
```
- Add local host alias `kafka.local` to your machines `/etc/hosts` file. You will need to use sudo access to edit this file.
- Run example in sbt, for example:
```
examples/runMain example3.CatsEffectApp
```