{"id":15221514,"url":"https://github.com/googlecloudplatform/dataflow-solution-guides","last_synced_at":"2025-10-20T00:32:28.315Z","repository":{"id":243642954,"uuid":"803969540","full_name":"GoogleCloudPlatform/dataflow-solution-guides","owner":"GoogleCloudPlatform","description":"The Dataflow Solution Guides offer full end-to-end deployment for the most common streaming solutions to run on Dataflow.","archived":false,"fork":false,"pushed_at":"2025-03-24T14:47:37.000Z","size":5705,"stargazers_count":31,"open_issues_count":17,"forks_count":15,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-19T22:18:17.667Z","etag":null,"topics":["beam","dataflow","gcp","google-cloud","google-cloud-platform","java","python","terraform"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GoogleCloudPlatform.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-21T17:52:11.000Z","updated_at":"2025-03-24T14:47:42.000Z","dependencies_parsed_at":"2024-06-10T11:44:21.483Z","dependency_job_id":"9651bbac-ecaa-40b8-ab92-6fd03fdeea38","html_url":"https://github.com/GoogleCloudPlatform/dataflow-solution-guides","commit_stats":null,"previous_names":["googlecloudplatform/dataflow-solution-guides"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/GoogleCloudPlatform/dataflow-solution-guides","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fdataflow-solution-guides","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fdataflow-solution-guides/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fdataflow-solution-guides/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fdataflow-solution-guides/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GoogleCloudPlatform","download_url":"https://codeload.github.com/GoogleCloudPlatform/dataflow-solution-guides/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fdataflow-solution-guides/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279955501,"owners_count":26250508,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-19T02:00:07.647Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beam","dataflow","gcp","google-cloud","google-cloud-platform","java","python","terraform"],"created_at":"2024-09-28T15:05:28.720Z","updated_at":"2025-10-20T00:32:25.322Z","avatar_url":"https://github.com/GoogleCloudPlatform.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dataflow Solution Guides\n\n[![Build and validation](https://github.com/GoogleCloudPlatform/dataflow-solution-guides/actions/workflows/pull_request.yml/badge.svg)](https://github.com/GoogleCloudPlatform/dataflow-solution-guides/actions/workflows/pull_request.yml) [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)\n\nWelcome to the Dataflow Solution Guides!\n\nThe Dataflow Solution Guides offer full end-to-end deployment for the most\ncommon streaming solutions to run\non [Dataflow](https://cloud.google.com/dataflow/).\n\nThis repository contains the following assets for each guide:\n\n- Full Terraform code to spawn all the necessary Google Cloud infrastructure\n- Pipelines code in Python, Java and Go (coming soon) for a\n  sample pipeline for each use case\n\n## Solution guides\n\nThis the list of solution guides available at this moment:\n\n|                              Guide                              |                                              Description                                               |    Development status     |\n| :-------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------: | :-----------------------: |\n|  [GenAI \u0026 Machine Learning Inference](./use_cases/GenAI_ML.md)  |                        Real-time inference with local GenAI models, using a GPU                        | Ready :white_check_mark:  |\n|       [ETL / Integration](./use_cases/ETL_integration.md)       | Real-time change data capture from a Spanner database to BigQuery                                      | Ready :white_check_mark:  |\n|  [Log Replication \u0026 Analytics](./use_cases/Log_replication.md)  |                                Real-time log replication into Splunk                                   |      Beta :factory:       |\n| [Marketing Intelligence](./use_cases/Marketing_Intelligence.md) |               Real-time marketing intelligence, using an AutoML model deployed in Vertex               |      Beta :factory:       |\n|  [Clickstream Analytics](./use_cases/Clickstream_Analytics.md)  |               Real-time clickstream analytics with Bigtable enrichment / data hydration                | Work in progress :hammer: |\n| [IoT Analytics](./use_cases/IoT_Analytics.md)                   |  Real-time Internet of Things (IoT) analytics with Bigtable enrichment \u0026 models deployed in Vertex AI  | Work in progress :hammer: |\n|      [Anomaly Detection](./use_cases/Anomaly_Detection.md)      |Real-time detection of anomalies in a stream of data leveraging GenAI with models deployed in Vertex AI |      Beta :factory:       |\n|          [Customer Data Platform](./use_cases/CDP.md)           |         Real-time customer data platform that unifies a customer view from different sources.          |      Beta :factory:       |\n|      [Gaming Analytics](./use_cases/gaming_analytics.md)        |          Real-time analyis of gaming data to enhance live gameplay \u0026 offer targeting                   |      Beta :factory:       |\n\n\n\n## Repository structure\n\n- `terraform`: This directory contains the Terraform code for deploying the\n  necessary Google Cloud\n  infrastructure for each use case.\n- `pipelines`: This directory contains the Python, Java, and Go code for the\n  sample pipelines.\n- `use_cases`: This directory contains the documentation of each use case\n\n## Getting help\n\n- GitHub Issues: Report any issues or ask questions on the GitHub repository.\n  - https://github.com/GoogleCloudPlatform/dataflow-solution-guides/issues\n- Stack Overflow: Search for existing solutions or ask questions on Stack\n  Overflow using the `google-cloud-dataflow` tag:\n  - https://stackoverflow.com/questions/tagged/google-cloud-dataflow\n\n## Contributing\n\nYour contributions to this repository are welcome.\n\n- Fork and Pull Request: Fork the repository and submit a pull request with your\n  changes.\n- Follow the Contribution Guidelines: Please follow the contribution guidelines\n  outlined in the\n  [CONTRIBUTING.md](CONTRIBUTING.md) file.\n\n## Disclaimer\n\nThis is not an officially supported Google product. The code in this repository\nis for demonstrative purposes only.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgooglecloudplatform%2Fdataflow-solution-guides","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgooglecloudplatform%2Fdataflow-solution-guides","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgooglecloudplatform%2Fdataflow-solution-guides/lists"}