{"id":22480157,"url":"https://github.com/findify/flink-adt","last_synced_at":"2025-05-01T09:17:50.169Z","repository":{"id":38022500,"uuid":"207827409","full_name":"findify/flink-adt","owner":"findify","description":"Scala ADT support for Apache Flink","archived":false,"fork":false,"pushed_at":"2024-07-29T14:47:22.000Z","size":207,"stargazers_count":52,"open_issues_count":26,"forks_count":9,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-05-01T09:17:26.250Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/findify.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-11T14:05:28.000Z","updated_at":"2025-02-24T10:10:26.000Z","dependencies_parsed_at":"2025-01-03T09:10:47.053Z","dependency_job_id":"b99fa3b0-2952-421e-a103-98446b661756","html_url":"https://github.com/findify/flink-adt","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findify%2Fflink-adt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findify%2Fflink-adt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findify%2Fflink-adt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findify%2Fflink-adt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/findify","download_url":"https://codeload.github.com/findify/flink-adt/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251850182,"owners_count":21653978,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-06T15:19:56.704Z","updated_at":"2025-05-01T09:17:50.122Z","avatar_url":"https://github.com/findify.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Scala ADT support for Apache Flink\n\n[![CI Status](https://github.com/findify/flink-adt/workflows/CI/badge.svg)](https://github.com/findify/flink-adt/actions)\n[![Maven Central](https://maven-badges.herokuapp.com/maven-central/io.findify/flink-adt_2.12/badge.svg?style=plastic)](https://maven-badges.herokuapp.com/maven-central/io.github.metarank/cfor_2.13)\n[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)\n\nThis is a prototype of Magnolia-based serializer framework for Apache Flink, with\nmore Scala-specific TypeSerializer \u0026 TypeInformation derivation support.\n\n* can support ADTs (Algebraic data types, sealed trait hierarchies)\n* correctly handles `case object` \n* can be extended with custom serializers even for deeply-nested types, as it uses implicitly available serializers\n  in the current scope\n* has no silent fallback to Kryo: it will just fail the compilation in a case when serializer cannot be made\n* reuses all the low-level serialization code from Flink for basic Java and Scala types\n* supports Scala 2.12, 2.13 \u0026 3.\n* built for scala-free Flink 1.15, which supports arbitrary Scala versions.\n\nIssues:\n* as this project relies on macro to derive TypeSerializer instances, if you're using IntelliJ 2020.*, it may\nhighlight your code with red, hinting that it cannot find corresponding implicits. And this is fine, the code\ncompiles OK. 2022.1 is fine with serializers derived with this library.\n  \n## Usage\n\n`flink-adt` is released to Maven-central. For SBT, add this snippet to `build.sbt`:\n```scala\nlibraryDependencies += \"io.findify\" %% \"flink-adt\" % \"0.6.1\"\n```\n\nTo use this library, swap `import org.apache.flink.api.scala._` with `import io.findify.flinkadt.api._` and enjoy.\n\nSo to derive a TypeInformation for a sealed trait, you can do:\n```scala\nimport io.findify.flinkadt.api._\nimport org.apache.flink.api.common.typeinfo.TypeInformation\n\nsealed trait Event extends Product with Serializable\n\nobject Event {\n  final case class Click(id: String) extends Event\n  final case class Purchase(price: Double) extends Event\n\n  implicit val eventTypeInfo: TypeInformation[Event] = deriveTypeInformation\n}\n```\n\nBe careful with a wildcard import of `import org.apache.flink.api.scala._`: it has a `createTypeInformation` implicit\nfunction, which may happily generate you a kryo-based serializer in a place you never expected. So in a case if you want\nto do this type of wildcard import, make sure that you explicitly called `deriveTypeInformation`\nfor all the sealed traits in the current scope.\n\n## Java types\n\nflink-adt is a scala-specific library and won't derive TypeInformation for java classes (as they don't extend the `scala.Product` type).\nBut you can always fall back to flink's own POJO serializer in this way, so just make it implicit so flink-adt can pick it up:\n\n```scala\nimport java.date.LocalDate\nimplicit val localDateTypeInfo: TypeInformation[LocalDate] = TypeInformation.of(classOf[LocalDate])\n```\n\n## Type mapping\n\nSometimes flink-adt may spot a type (usually a java one), which cannot be directly serialized as a case class, like this \nexample:\n```scala\n  class WrappedString {\n    private var internal: String = \"\"\n\n    override def equals(obj: Any): Boolean = obj match {\n      case s: WrappedString =\u003e s.get == internal\n      case _                =\u003e false\n    }\n    def get: String = internal\n    def put(value: String) = {\n      internal = value\n    }\n  }\n```\n\nYou can write a pair of explicit `TypeInformation[WrappedString]` and `Serializer[WrappedString]`, but it's extremely verbose,\nand the class itself can be 1-to-1 mapped to a regular `String`. Flink-adt has a mechanism of type mappers to delegate serialization\nof non-serializable types to existing serializers. For example:\n```scala\n  class WrappedMapper extends TypeMapper[WrappedString, String] {\n    override def map(a: WrappedString): String = a.get\n\n    override def contramap(b: String): WrappedString = {\n      val str = new WrappedString\n      str.put(b)\n      str\n    }\n  }\n  implicit val mapper: TypeMapper[WrappedString, String] = new WrappedMapper()\n  // will treat WrappedString with String typeinfo:\n  implicit val ti: TypeInformation[WrappedString] = implicitly[TypeInformation[WrappedString]] \n```\n\nWhen there is a `TypeMapper[A,B]` in the scope to convert `A` to `B` and back, and type `B` has `TypeInformation[B]` available \nin the scope also, then flink-adt will use a delegated existing typeinfo for `B` when it will spot type `A`.\n\nWarning: on Scala 3, the TypeMapper should not be made anonymous. This example won't work, as anonymous implicit classes in \nscala 3 are private, and Flink cannot instantiate it on restore without jvm17 incompatible reflection hacks:\n```scala\n  // anonymous class, will fail on runtime on scala 3\n  implicit val mapper: TypeMapper[WrappedString, String] = new TypeMapper[WrappedString, String] {\n    override def map(a: WrappedString): String = a.get\n\n    override def contramap(b: String): WrappedString = {\n      val str = new WrappedString\n      str.put(b)\n      str\n    }\n  }\n```\n\n## Schema evolution\n\nFor the child case classes being part of ADT, `flink-adt` uses a Flink's `ScalaCaseClassSerializer`, so all the compatibility rules\nare the same as for normal case classes.\n\nFor the sealed trait membership itself, `flink-adt` used an own serialization format with the following rules:\n* you cannot reorder trait members, as wire format depends on the compile-time index of each member\n* you can add new members at the end of the list\n* you cannot remove ADT members\n* you cannot replace ADT members\n\n## Compatibility\n\nThis project uses a separate set of serializers for collections, instead of Flink's own TraversableSerializer. So probably you\nmay have issues while migrating state snapshots from TraversableSerializer to FlinkADT ones.\n\nStarting from version 0.5.0+, this project strictly depends on Flink 1.15+, as starting from this version it can be cross-built \nfor scala 2.13 and scala 3.\n\n## Licence\n\nThe MIT License (MIT)\n\nCopyright (c) 2022 Findify AB\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffindify%2Fflink-adt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffindify%2Fflink-adt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffindify%2Fflink-adt/lists"}