{"id":50350502,"url":"https://github.com/cchenax/streamforge-ai","last_synced_at":"2026-06-15T12:01:40.477Z","repository":{"id":345760319,"uuid":"1184291466","full_name":"cchenax/streamforge-ai","owner":"cchenax","description":"Open-source real-time AI data pipeline for CDC ingestion, feature generation, and storage-aware prefetching","archived":false,"fork":false,"pushed_at":"2026-06-06T05:10:44.000Z","size":20084,"stargazers_count":307,"open_issues_count":10,"forks_count":28,"subscribers_count":26,"default_branch":"main","last_synced_at":"2026-06-06T07:07:21.787Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cchenax.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-17T12:54:43.000Z","updated_at":"2026-06-06T05:10:49.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/cchenax/streamforge-ai","commit_stats":null,"previous_names":["cchenax/streamforge-ai"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cchenax/streamforge-ai","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cchenax%2Fstreamforge-ai","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cchenax%2Fstreamforge-ai/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cchenax%2Fstreamforge-ai/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cchenax%2Fstreamforge-ai/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cchenax","download_url":"https://codeload.github.com/cchenax/streamforge-ai/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cchenax%2Fstreamforge-ai/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34361403,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-15T02:00:07.085Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-29T21:00:23.920Z","updated_at":"2026-06-15T12:01:40.471Z","avatar_url":"https://github.com/cchenax.png","language":"Python","funding_links":[],"categories":["大数据"],"sub_categories":["微服务框架"],"readme":"\n\n# StreamForge AI\n\n## 1. Project Overview\n\nStreamForge AI is a real-time data pipeline platform for AI and analytics workloads. It focuses on:\n\n- CDC ingestion from operational databases\n- stream processing for feature generation\n- object-storage-based data sinking\n- storage-aware prefetching for ML workloads\n\n## 2. Motivation\n\n- Provide a minimal but realistic open-source AI data pipeline\n- Support local development and demo environments\n- Showcase best practices in streaming, storage, and pipeline orchestration\n- Demonstrate architecture leadership and contributor collaboration\n\n## 3. Non-goals\n\n- Full production-grade multi-tenant platform\n- Large-scale distributed control plane\n- Enterprise authentication / authorization in v0.1\n\n## Architecture Summary\n\n### 4.1 Ingestion layer\nUses Debezium to capture row-level changes from MySQL/Postgres and publish them to Kafka topics.\n\n### 4.2 Streaming layer\nUses Apache Flink to:\n- consume CDC events\n- perform cleaning and transformation\n- compute simple feature aggregations\n- write processed outputs to storage\n\n### 4.3 Storage layer\nMinIO/S3-compatible storage is the initial storage target (and is optionally exercised by the prefetch demo).\nFuture versions may support Iceberg table sinks for incremental analytics.\n\n### 4.4 Prefetch layer\nA lightweight prefetch engine analyzes expected access patterns and pulls selected objects into a hot cache area before an ML job starts (implemented in `prefetch-engine/`).\n\n## 5. Initial module boundaries\n\n- `stream-processor/`\n- `prefetch-engine/`\n- `deploy/`\n\n## 6. Design decisions\n\n### Why Kafka\nKafka is a widely adopted event backbone and works naturally with Debezium and Flink.\n\n### Why Flink\nFlink provides strong streaming semantics, checkpointing, and event processing flexibility.\n\n### Why MinIO first\nMinIO is simple for local demos and provides an S3-compatible interface.\n\n### Why prefetch demo\nPrefetching is a practical optimization for AI pipelines with repeated object access and training cold starts.\n\n## Core Features\n\nThis repo focuses on an MVP demo set that illustrates the intended architecture:\n- MySQL -\u003e Kafka CDC ingestion via Debezium (see `deploy/cdc-mysql-kafka-debezium/`)\n- End-to-end demo: MySQL -\u003e Debezium -\u003e Kafka -\u003e Flink -\u003e MinIO (see `deploy/cdc-flink-minio-demo/`)\n- Storage-aware prefetching demo for ML workloads (see `prefetch-engine/`)\n- Optional MinIO upload of processed outputs from the prefetch demo (see `prefetch-engine/README.md`)\n\nPlanned next:\n- Additional storage sinks (e.g., Iceberg)\n- Metrics and observability\n- Benchmark scenarios\n\n## Roadmap\n\n### v0.1 Local demo (MVP)\n- [x] Storage-aware prefetch demo (`prefetch-engine/`)\n- [x] MySQL -\u003e Kafka CDC ingestion via Debezium (`deploy/cdc-mysql-kafka-debezium/`)\n- [x] End-to-end demo: MySQL -\u003e Debezium -\u003e Kafka -\u003e Flink -\u003e MinIO (`deploy/cdc-flink-minio-demo/`)\n- [x] Runnable local stack (Docker Compose) for the full demo (`deploy/cdc-flink-minio-demo/docker-compose.yml`)\n\n### v0.2 Streaming + sinks\n- [x] Flink stream processor job + example (`stream-processor/`)\n- [x] MinIO/S3 sink with an output layout and naming conventions (`deploy/cdc-flink-minio-demo/feature-sink/`)\n- [x] Iceberg sink support (optional)\n\n### v0.3 Hardening + integrations\n- [x] Schema evolution handling\n- [x] Metrics and observability\n- [x] Benchmark scenarios\n- [x] Training-job integration example\n- [x] Backfill / replay tooling for historical reprocessing\n- [x] Data quality checks (basic validation + drift signals)\n- [x] Cost/performance tuning guide (Flink, Kafka, MinIO, Iceberg)\n\n### v0.4 Lakehouse + governance\n- [x] Iceberg-first sink mode (partitioning, compaction, snapshots)\n- [x] Table/catalog integration (REST catalog or Hive Metastore)\n- [x] Data lineage basics (job -\u003e dataset -\u003e feature artifacts)\n- [x] Access patterns for offline/online features (example layouts)\n\n### v0.5 Platformization\n- [x] Pipeline configuration as code (YAML) + validation\n- [x] Simple control-plane API for starting/stopping pipelines\n- [x] Web UI for demo environments (status, logs, artifacts)\n- [x] Basic authn/authz for local multi-user demos\n\n### v0.6 Expanded sources \u0026 feature serving\n- [ ] Multi-source CDC support (PostgreSQL, MongoDB)\n- [ ] Feature store integration (Feast, Tecton-compatible interface)\n- [ ] Online feature serving API with caching\n- [ ] Schema registry integration (Confluent Schema Registry or Apicurio)\n- [ ] Stream-to-batch joins (Flink with Iceberg dimension tables)\n\n### v0.7 ML/AI deeper integration\n- [ ] Real-time feature drift detection \u0026 alerting\n- [ ] Streaming windowed feature examples (rolling aggregates, sessionization)\n- [ ] Real-time inference pipeline example (Flink + model server)\n- [ ] Training data sampling/downsampling for active learning\n- [ ] Metadata catalog for features \u0026 pipelines (OpenLineage/Marquez)\n\n### v0.8 Enterprise readiness\n- [ ] Kubernetes operator for cluster deployment\n- [ ] Secrets management (Vault integration or Kubernetes Secrets)\n- [ ] Fine-grained RBAC for pipelines/datasets\n- [ ] Audit logging (API, job, and data access events)\n- [ ] High-availability mode for control plane \u0026 critical services\n\n### v0.9 AI/RAG \u0026 agent workloads\n- [ ] First-class RAG pipeline template (ingest → embed → index → retrieve) on top of `rag-engine/`\n- [ ] Pluggable embedding model adapters (local, OpenAI-compatible, HuggingFace) with batching\n- [ ] Vector store sinks (pgvector, Qdrant, Milvus) alongside existing Iceberg/MinIO sinks\n- [ ] Hybrid retrieval (BM25 + vector) with reranker hooks\n- [ ] Agent workflow orchestration examples (`agent-workflow/`) wired to streaming features\n- [ ] Tool/function-calling integration with control-plane API for pipeline actions\n- [ ] Streaming-to-RAG freshness SLOs (latency/lag dashboards per index)\n- [ ] LLM-output evaluation harness (groundedness, hallucination, retrieval recall) + benchmarks\n- [ ] PII/safety filters on ingestion and retrieval paths\n- [ ] Cost/token telemetry for embedding + inference per pipeline\n\n### Roadmap Links\n- GitHub issues: https://github.com/cchenax/streamforge-ai/issues\n- GitHub projects (if/when populated): https://github.com/cchenax/streamforge-ai/projects\n\n## Contribution Links\n- Contribution guide: [`CONTRIBUTING.md`](./CONTRIBUTING.md)\n- Start a topic/track progress: https://github.com/cchenax/streamforge-ai/issues/new/choose\n- Open PRs to `main`: https://github.com/cchenax/streamforge-ai/pulls\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcchenax%2Fstreamforge-ai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcchenax%2Fstreamforge-ai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcchenax%2Fstreamforge-ai/lists"}