https://github.com/irwandifo/gcp-batch-infra
GCP Infrastructure for Batch Processing
https://github.com/irwandifo/gcp-batch-infra
data-lakehouse data-platform gcp terraform
Last synced: 20 days ago
JSON representation
GCP Infrastructure for Batch Processing
- Host: GitHub
- URL: https://github.com/irwandifo/gcp-batch-infra
- Owner: irwandifo
- Created: 2025-01-07T23:12:47.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-02-11T01:44:48.000Z (over 1 year ago)
- Last Synced: 2025-10-30T10:34:35.918Z (9 months ago)
- Topics: data-lakehouse, data-platform, gcp, terraform
- Language: HCL
- Homepage:
- Size: 1.39 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# GCP Infrastructure for Batch Processing
This project sets up a lightweight and cost-effective batch processing infrastructure for data platform on GCP. It is designed for small-to-medium-scale data processing tasks, leveraging GCP services like Cloud SQL, VPC, Compute Engine, Cloud Storage, and BigQuery.
## Components
| **Categories** | **Tools** | **Details** |
|------------------------|-----------------|-------------------------------------------------|
| Infrastructure-as-code | Terraform | Automates infrastructure management. |
| Database | Cloud SQL | Simulates PostgreSQL OLTP database. |
| Networking | VPC | Secure Cloud SQL to Compute Engine Networking. |
| Processing | Compute Engine | Deploy kestra to process workflows. |
| Storage | Cloud Storage | Stores data in bronze and silver layer. |
| Consumption | BigQuery | Stores data in gold/consumption layer. |
## Architecture Diagram
