https://github.com/datadome/public-flink-utils
Utilities for Flink
https://github.com/datadome/public-flink-utils
flink public
Last synced: about 1 month ago
JSON representation
Utilities for Flink
- Host: GitHub
- URL: https://github.com/datadome/public-flink-utils
- Owner: DataDome
- License: apache-2.0
- Created: 2020-10-07T12:21:47.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2022-11-08T08:28:21.000Z (over 3 years ago)
- Last Synced: 2025-02-28T12:21:39.381Z (over 1 year ago)
- Topics: flink, public
- Language: Scala
- Homepage:
- Size: 20.5 KB
- Stars: 2
- Watchers: 6
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
This is a **PUBLIC** repository for some tools we use in our Flink jobs.
# State signature
This makes the UID of an operator change automatically whenever there is some change in the state structure, making sure Flink will not try to reload the old state.
In your model classes:
```scala
import co.datadome.flinkutils.util.TypeSignature
case class BotBuster(name: String)
object BotBuster {
implicit val typeSignature: TypeSignature[BotBuster] =
TypeSignature.forSimpleCaseClass[BotBuster]
}
class Something { … }
object Something {
implicit val typeSignature: TypeSignature[Something] =
TypeSignature[Something]
.withType[String]
.withType[BotBuster]
.build
}
```
In your operator:
```scala
import co.datadome.flinkutils.util.{StateTypeSignature, TypeSignature}
import org.apache.flink.streaming.api.functions.co.KeyedBroadcastProcessFunction
class MyKeyedBroadcastProcessFunction extends KeyedBroadcastProcessFunction[Int, String, Boolean, Int] { … }
object MyKeyedBroadcastProcessFunction {
val stateSignature = StateTypeSignature.forKeyedBroadcastFunction[MyKeyedBroadcastProcessFunction]
.withKey[Int]
.withValueState[Set[String]]
.withBroadcastState[Int, BotBuster]
.build
}
```
In your job:
```scala
import org.apache.flink.streaming.api.scala._
import co.datadome.flinkutils.syntax._
val stream: DataStream[String] = ???
stream.uidNameStated[MyKeyedBroadcastProcessFunction]("my-function")
```