Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/googlecloudplatform/dataflow-solution-guides

The Dataflow Solution Guides offer full end-to-end deployment for the most common streaming solutions to run on Dataflow.
https://github.com/googlecloudplatform/dataflow-solution-guides

beam dataflow gcp google-cloud google-cloud-platform java python terraform

Last synced: 3 days ago
JSON representation

The Dataflow Solution Guides offer full end-to-end deployment for the most common streaming solutions to run on Dataflow.

Host: GitHub
URL: https://github.com/googlecloudplatform/dataflow-solution-guides
Owner: GoogleCloudPlatform
License: apache-2.0
Created: 2024-05-21T17:52:11.000Z (9 months ago)
Default Branch: main
Last Pushed: 2025-02-01T22:25:18.000Z (6 days ago)
Last Synced: 2025-02-01T23:24:12.566Z (6 days ago)
Topics: beam, dataflow, gcp, google-cloud, google-cloud-platform, java, python, terraform
Language: HCL
Homepage:
Size: 5.37 MB
Stars: 29
Watchers: 1
Forks: 14
Open Issues: 6
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE

Awesome Lists containing this project

README

        # Dataflow Solution Guides

[![Build and validation](https://github.com/GoogleCloudPlatform/dataflow-solution-guides/actions/workflows/pull_request.yml/badge.svg)](https://github.com/GoogleCloudPlatform/dataflow-solution-guides/actions/workflows/pull_request.yml) [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)

Welcome to the Dataflow Solution Guides!

The Dataflow Solution Guides offer full end-to-end deployment for the most

common streaming solutions to run

on [Dataflow](https://cloud.google.com/dataflow/).

This repository contains the following assets for each guide:

- Full Terraform code to spawn all the necessary Google Cloud infrastructure

- Pipelines code in Python, Java and Go (coming soon) for a

  sample pipeline for each use case

## Solution guides

This the list of solution guides available at this moment:

|                              Guide                              |                                              Description                                               |    Development status     |

| :-------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------: | :-----------------------: |

|  [GenAI & Machine Learning Inference](./use_cases/GenAI_ML.md)  |                        Real-time inference with local GenAI models, using a GPU                        | Ready :white_check_mark:  |

|       [ETL / Integration](./use_cases/ETL_integration.md)       | Real-time change data capture from a Spanner database to BigQuery                                      | Ready :white_check_mark:  |

|  [Log Replication & Analytics](./use_cases/Log_replication.md)  |                                Real-time log replication into Splunk                                   |      Beta :factory:       |

| [Marketing Intelligence](./use_cases/Marketing_Intelligence.md) |               Real-time marketing intelligence, using an AutoML model deployed in Vertex               |      Beta :factory:       |

|  [Clickstream Analytics](./use_cases/Clickstream_Analytics.md)  |               Real-time clickstream analytics with Bigtable enrichment / data hydration                | Work in progress :hammer: |

| [IoT Analytics](./use_cases/IoT_Analytics.md)                   |  Real-time Internet of Things (IoT) analytics with Bigtable enrichment & models deployed in Vertex AI  | Work in progress :hammer: |

|      [Anomaly Detection](./use_cases/Anomaly_Detection.md)      |Real-time detection of anomalies in a stream of data leveraging GenAI with models deployed in Vertex AI |      Beta :factory:       |

|          [Customer Data Platform](./use_cases/CDP.md)           |         Real-time customer data platform that unifies a customer view from different sources.          |      Beta :factory:       |

|      [Gaming Analytics](./use_cases/gaming_analytics.md)        |          Real-time analyis of gaming data to enhance live gameplay & offer targeting                   |      Beta :factory:       |

## Repository structure

- `terraform`: This directory contains the Terraform code for deploying the

  necessary Google Cloud

  infrastructure for each use case.

- `pipelines`: This directory contains the Python, Java, and Go code for the

  sample pipelines.

- `use_cases`: This directory contains the documentation of each use case

## Getting help

- GitHub Issues: Report any issues or ask questions on the GitHub repository.

  - https://github.com/GoogleCloudPlatform/dataflow-solution-guides/issues

- Stack Overflow: Search for existing solutions or ask questions on Stack

  Overflow using the `google-cloud-dataflow` tag:

  - https://stackoverflow.com/questions/tagged/google-cloud-dataflow

## Contributing

Your contributions to this repository are welcome.

- Fork and Pull Request: Fork the repository and submit a pull request with your

  changes.

- Follow the Contribution Guidelines: Please follow the contribution guidelines

  outlined in the

  [CONTRIBUTING.md](CONTRIBUTING.md) file.

## Disclaimer

This is not an officially supported Google product. The code in this repository

is for demonstrative purposes only.