{"id":15500390,"url":"https://github.com/eliias/gleam","last_synced_at":"2025-10-18T19:05:16.183Z","repository":{"id":70227061,"uuid":"289132196","full_name":"eliias/gleam","owner":"eliias","description":"Fun DSL for Apache Beam and Kotlin.","archived":false,"fork":false,"pushed_at":"2023-01-02T14:02:13.000Z","size":2852,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-10T23:24:24.648Z","etag":null,"topics":["apache-beam","data-engineering","stream-processing"],"latest_commit_sha":null,"homepage":"","language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eliias.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-20T23:32:22.000Z","updated_at":"2024-03-27T12:19:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"2b0fd60a-3bd9-4e95-ba7f-c7d58644dad8","html_url":"https://github.com/eliias/gleam","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/eliias/gleam","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eliias%2Fgleam","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eliias%2Fgleam/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eliias%2Fgleam/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eliias%2Fgleam/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eliias","download_url":"https://codeload.github.com/eliias/gleam/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eliias%2Fgleam/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267988397,"owners_count":24176996,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-31T02:00:08.723Z","response_time":66,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-beam","data-engineering","stream-processing"],"created_at":"2024-10-02T09:00:08.624Z","updated_at":"2025-10-18T19:05:11.150Z","avatar_url":"https://github.com/eliias.png","language":"Kotlin","funding_links":[],"categories":[],"sub_categories":[],"readme":"# gleam\n\n\u003e Fun with Apache Beam.\n\n## gleam-core\n\nExtends the Apache Beam library with convenience functions, mostly to better\nutilize type inference in Kotlin.\n\n### `transform`\n\nA convenience function to create a new transform with one or more steps.\n\n```kotlin\nval newTransform = transform {\n  it.apply(SomeTransform())\n}\n```\n\nA default implementation without this helper would look more like this:\n\n```kotlin\nval newTransform = object : PTransform\u003cPCollection\u003cT\u003e, POutput\u003e() {\n  override fun expand(input: PCollection\u003cT\u003e) = input.apply(SomeTransform)\n}\n```\n\n## gleam-flow\n\nA DSL that utilizes Generics, Scope Functions, and Type Inference to provide the\ntransforms from the Apache Beam library.\n\n```kotlin\nval pipeline = Pipeline.create()\n\npipeline\n  .sequence()\n  .window(FixedWindows.of(Duration.standardSeconds(1)))\n  .map { it * 2 }\n  .filter { it % 2 == 0L }\n  .key { it / 10 }\n  .groupByKey()\n  .sink(Log.info())\n\npipeline.run()\n```\n\n## gleam-samples\n\n### 1. Batch\n\nThis demo's batching of all elements of a bundle. This is a very useful\ntechnique for IO heavy DoFn's.\n\nExample: An upstream transform provides a collection of record keys, and the\ndownstream transform needs to fetch the records from a persistent datastore, it\nis usually better to batch multiple keys into a single request.\n\n### 2. DSL\n\nA Kotlin specific Apache Beam DSL that allows applying transforms in a direct\nway onto collections.\n\n### 3. Schema Extensions\n\nAdd metadata to schemas or single fields. The metadata and hints are stored as\nschema options. Directly with the schema and field definition.\n\n```kotlin\nval schema = schema {\n  version(\"1.0\")\n\n  // base\n  field(\"id\", INT64.withNullable(true)) {\n    // adds the \"version\" extension\n    version(\"1.0\")\n  }\n  fields(\n    Field.of(\"title\", STRING),\n    Field.of(\"description\", STRING)\n  ) {\n    // adds the \"compress\" extension to all fields!\n    compress()\n  }\n}\n```\n\n### 4. POJO's\n\nDemo's the usage of POJO's (Beans) and field schemas in Beam with Kotlin\n\n### 6. SDF\n\nA running version of the SDF example from the Apache Beam website.\n\n### 14. Resources\n\nInversion of Control and Dependency Injection with Apache Beam, Kotlin and Koin.\n\n### 20. Generator\n\nA demo of the `gleam-core` Generator to create fake data for a pipeline.\n\n### 23. Bigtable\n\nA demo of the `BigtableIO.write` transform.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feliias%2Fgleam","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feliias%2Fgleam","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feliias%2Fgleam/lists"}