https://github.com/datadome/public-flink-utils

Utilities for Flink
https://github.com/datadome/public-flink-utils

flink public

Last synced: 10 days ago
JSON representation

Utilities for Flink

• Host: GitHub
• URL: https://github.com/datadome/public-flink-utils
• Owner: DataDome
• License: apache-2.0
• Created: 2020-10-07T12:21:47.000Z (over 4 years ago)
• Default Branch: master
• Last Pushed: 2022-11-08T08:28:21.000Z (about 2 years ago)
• Last Synced: 2024-11-12T05:33:06.643Z (2 months ago)
• Topics: flink, public
• Language: Scala
• Homepage:
• Size: 20.5 KB
• Stars: 2
• Watchers: 6
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE
  • Codeowners: .github/CODEOWNERS

Awesome Lists containing this project

README

        
This is a **PUBLIC** repository for some tools we use in our Flink jobs.

# State signature

This makes the UID of an operator change automatically whenever there is some change in the state structure, making sure Flink will not try to reload the old state.

In your model classes:

```scala

  import co.datadome.flinkutils.util.TypeSignature

  case class BotBuster(name: String)

  object BotBuster {

    implicit val typeSignature: TypeSignature[BotBuster] =

      TypeSignature.forSimpleCaseClass[BotBuster]

  }

  class Something { … }

  object Something {

    implicit val typeSignature: TypeSignature[Something] =

      TypeSignature[Something]

        .withType[String]

        .withType[BotBuster]

        .build

  }

```

In your operator:

```scala

  import co.datadome.flinkutils.util.{StateTypeSignature, TypeSignature}

  import org.apache.flink.streaming.api.functions.co.KeyedBroadcastProcessFunction

  class MyKeyedBroadcastProcessFunction extends KeyedBroadcastProcessFunction[Int, String, Boolean, Int] { … }

  object MyKeyedBroadcastProcessFunction {

    val stateSignature = StateTypeSignature.forKeyedBroadcastFunction[MyKeyedBroadcastProcessFunction]

      .withKey[Int]

      .withValueState[Set[String]]

      .withBroadcastState[Int, BotBuster]

      .build

  }

```

In your job:

```scala

  import org.apache.flink.streaming.api.scala._

  import co.datadome.flinkutils.syntax._

  val stream: DataStream[String] = ???

  stream.uidNameStated[MyKeyedBroadcastProcessFunction]("my-function")

```