{"id":22480151,"url":"https://github.com/findify/flink-protobuf","last_synced_at":"2025-08-02T14:32:43.527Z","repository":{"id":45803644,"uuid":"372549162","full_name":"findify/flink-protobuf","owner":"findify","description":"Protobuf serialization support for Apache Flink","archived":false,"fork":false,"pushed_at":"2021-06-01T15:21:02.000Z","size":29,"stargazers_count":19,"open_issues_count":1,"forks_count":5,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-05-02T23:38:22.687Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/findify.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-05-31T15:17:17.000Z","updated_at":"2024-04-30T13:28:32.000Z","dependencies_parsed_at":"2022-07-17T12:16:22.742Z","dependency_job_id":null,"html_url":"https://github.com/findify/flink-protobuf","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findify%2Fflink-protobuf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findify%2Fflink-protobuf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findify%2Fflink-protobuf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findify%2Fflink-protobuf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/findify","download_url":"https://codeload.github.com/findify/flink-protobuf/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228483524,"owners_count":17927363,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-06T15:19:54.224Z","updated_at":"2024-12-06T15:19:54.706Z","avatar_url":"https://github.com/findify.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Protobuf serialization support for Apache Flink\n\n[![CI Status](https://github.com/findify/flink-protobuf/workflows/CI/badge.svg)](https://github.com/flink-protobuf/workflows/actions)\n[![Maven Central](https://maven-badges.herokuapp.com/maven-central/io.findify/flink-protobuf_2.12/badge.svg?style=plastic)](https://maven-badges.herokuapp.com/maven-central/io.findify/flink-protobuf_2.12)\n[![License: Apache 2](https://img.shields.io/badge/License-Apache2-green.svg)](https://opensource.org/licenses/Apache2.0)\n\nThis project is an adapter to connect [Google Protobuf](https://developers.google.com/protocol-buffers) to the flink's \nown `TypeInformation`-based [serialization framework](https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html). \nThis project can be useful if you have:\n* [oneof-encoded](https://developers.google.com/protocol-buffers/docs/proto#oneof) protobuf messages, \n  which cannot be efficiently encoded using flink's serialization without Kryo fallback.\n* flexible requirements on schema evolution for POJO classes (as compared to \n  [Flinks' for POJOs and Scala case classes](https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/schema_evolution/))\n* schema evolution support is needed for scala case classes (as Flink lacks it out of the box)\n\n## Usage\n\n`flink-protobuf` is released to Maven-central. For SBT, add this snippet to `build.sbt`:\n```scala\nlibraryDependencies += \"io.findify\" %% \"flink-protobuf\" % \"0.2\"\n```\nThen, given that you have a following message format:\n```proto\nmessage Foo {\n    required int32 value = 1;\n}\n```\nYou can build a `TypeInformation` for scalapb-generated classes like this:\n```scala\nimport io.findify.flinkpb.FlinkProtobuf\n\nimplicit val ti = FlinkProtobuf.generateScala(Foo)\nval result      = env.fromCollection(List(Foo(1), Foo(2), Foo(3)))\n```\n\nFor Java it's going to look a bit different:\n```java\nimport io.findify.flinkprotobuf.java.Tests;\nimport org.apache.flink.api.common.typeinfo.TypeInformation;\nimport org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;\n\nTypeInformation\u003cTests.Foo\u003e ti = FlinkProtobuf.generateJava(Tests.Foo.class, Tests.Foo.getDefaultInstance());\nenv.fromCollection(List.of(Tests.Foo.newBuilder().setValue(1).build()), ti).executeAndCollect(100);\n\n```\n\n## Schema evolution\n\nCompared to Flink schema evolution for POJO classes, with `flink-protobuf` you can do much more:\n* fields can be renamed (as protobuf uses an index-based encoding for field names)\n* types can be changed (so optional field can be made repeated, or int32 can be upcasted to int64)\n\nFor Scala case classes Flink has no support for schema evolution, so with this project you can:\n* add, rename, remove fields\n* change field types\n\n## Compatibility\n\nThe library is built over Flink 1.13 for Scala 2.12, but should be binary compatible with older flink versions.\nScala 2.11 version is not planned, as ScalaPB already dropped it's support.\n\n## License\n\nApache License\nVersion 2.0, January 2004\nhttp://www.apache.org/licenses/","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffindify%2Fflink-protobuf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffindify%2Fflink-protobuf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffindify%2Fflink-protobuf/lists"}