https://github.com/mkuthan/stream-processing

Learn how to develop and test stateful streaming and batch data pipelines
https://github.com/mkuthan/stream-processing

apache-beam scio stream-processing

Last synced: about 2 months ago
JSON representation

Learn how to develop and test stateful streaming and batch data pipelines

• Host: GitHub
• URL: https://github.com/mkuthan/stream-processing
• Owner: mkuthan
• License: apache-2.0
• Created: 2022-01-23T13:55:34.000Z (almost 3 years ago)
• Default Branch: main
• Last Pushed: 2024-11-04T07:16:34.000Z (about 2 months ago)
• Last Synced: 2024-11-04T08:22:20.052Z (about 2 months ago)
• Topics: apache-beam, scio, stream-processing
• Language: Scala
• Homepage: http://mkuthan.github.io/tags/#stream-processing
• Size: 24.9 MB
• Stars: 6
• Watchers: 4
• Forks: 2
• Open Issues: 6
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Learn How To Develop And Test Stateful Streaming Data Pipelines

[![CI](https://github.com/mkuthan/stream-processing/actions/workflows/ci.yml/badge.svg)](https://github.com/mkuthan/stream-processing/actions/workflows/ci.yml)

[![codecov](https://codecov.io/gh/mkuthan/stream-processing/branch/main/graph/badge.svg?token=V9HUU6OJGF)](https://codecov.io/gh/mkuthan/stream-processing)

Shared modules:

* `stream-processing-shared` - shared utilities for developing stateful streaming data pipelines

* `stream-processing-infrastructure` - infrastructure layer with IOs for BigQuery, Pubsub and Cloud Storage

* `stream-processing-test` - shared utilities for testing stateful streaming data pipelines

Use cases:

* `toll-application`, `toll-domain`, `toll-infrastructure` - sample application for toll data processing,

see [blog post](https://mkuthan.github.io/blog/2023/09/27/unified-batch-streaming/)

* `word-count` - fixed window example,

see [blog post](https://mkuthan.github.io/blog/2022/01/28/stream-processing-part1/)

* `session-window` - session window example,

see [blog post](https://mkuthan.github.io/blog/2022/03/08/stream-processing-part2/)