{"id":51299915,"url":"https://github.com/pranavkakde/siphon","last_synced_at":"2026-06-30T18:02:49.574Z","repository":{"id":364128916,"uuid":"1265435552","full_name":"pranavkakde/siphon","owner":"pranavkakde","description":"Real-time open-source test analytics platform. Ingests low-latency gRPC test suite streams, buffers via RabbitMQ, sinks idempotently to MongoDB, and pipes metrics to a live WebSockets React dashboard.","archived":false,"fork":false,"pushed_at":"2026-06-11T18:04:53.000Z","size":753,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-11T20:05:02.519Z","etag":null,"topics":["clean-architecture","dashboard","devops-tools","golang","grpc","minio","mongodb","monorepo","observability","opentelemetry","rabbitmq","react","real-time","test-analytics","websocket"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pranavkakde.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-10T19:17:27.000Z","updated_at":"2026-06-11T18:04:58.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/pranavkakde/siphon","commit_stats":null,"previous_names":["pranavkakde/siphon"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/pranavkakde/siphon","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pranavkakde%2Fsiphon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pranavkakde%2Fsiphon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pranavkakde%2Fsiphon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pranavkakde%2Fsiphon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pranavkakde","download_url":"https://codeload.github.com/pranavkakde/siphon/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pranavkakde%2Fsiphon/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34977672,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-30T02:00:05.919Z","response_time":92,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clean-architecture","dashboard","devops-tools","golang","grpc","minio","mongodb","monorepo","observability","opentelemetry","rabbitmq","react","real-time","test-analytics","websocket"],"created_at":"2026-06-30T18:02:48.929Z","updated_at":"2026-06-30T18:02:49.568Z","avatar_url":"https://github.com/pranavkakde.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Siphon — Real-time Open-Source Test Analytics Platform\n\nSiphon is a high-throughput, real-time open-source test analytics platform designed to capture automated test suite execution metrics, manage screenshots of test failures via object storage, and stream live execution feeds straight to an interactive, premium engineering dashboard.\n\n![Siphon Dashboard Mockup](siphon_dashboard_mockup.png)\n\n---\n\n## Architecture \u0026 Communication Overview\n\nSiphon is built on a decoupled, microservice-oriented event pipeline. Here is how the UI and other services communicate with each other:\n\n```mermaid\ngraph TD\n  A[go-hook Client] --\u003e|gRPC Stream :50051| Ingress[siphon-ingress API]\n  A --\u003e|HTTP PUT :9000| MinIO[(MinIO Object Storage)]\n  \n  Ingress --\u003e|JSON over AMQP| Rabbit[RabbitMQ Buffer Queue]\n  Sink[siphon-sink Worker Pool] --\u003e|Consume| Rabbit\n  Sink --\u003e|Idempotent Upsert| DB[(MongoDB Time-Series)]\n  \n  StreamAPI[siphon-stream-api :8080] --\u003e|Poll tail / Change Stream| DB\n  \n  UI[siphon-glass UI :5173] --\u003e|1. WebSocket Connection :8080/ws| StreamAPI\n  UI --\u003e|2. HTTP GET :8080/api/runs \u0026 /api/stats| StreamAPI\n  UI --\u003e|3. HTTP GET :9000/siphon-screenshots/| MinIO\n```\n\n### Ingestion Pipeline (Source to Database)\n* **gRPC Streaming:** Test runner clients (like `go-hook`) stream binary test run data using Protocol Buffers directly to `siphon-ingress` on port `50051`.\n* **MinIO Storage:** Test clients upload error screenshot files directly to MinIO Object Storage (port `9000` / `9001`) via HTTP.\n* **Message Queue:** `siphon-ingress` serializes the incoming Protobuf payloads into JSON and publishes them to RabbitMQ (`siphon-buffer-queue`) to act as a buffer.\n* **Database Ingest:** The `siphon-sink` worker pool consumes messages from RabbitMQ and inserts/updates them idempotently into a MongoDB Time-Series collection.\n\n### UI Communication Details\nThe React UI (`siphon-glass`) interacts with backend services through three primary channels:\n1. **WebSocket Protocol (`ws://localhost:8080/ws`):** On application mount, the UI opens a persistent WebSocket connection to the `siphon-stream-api` service. As new test results land in MongoDB, the `siphon-stream-api` tailing thread detects them and broadcasts them in real time over the WebSocket channel. The UI receives these events and appends them dynamically to the **Activity Feed** without requiring a page refresh.\n2. **REST API Endpoint Requests (`http://localhost:8080/api/...`):**\n   * `GET /api/runs`: Used to fetch the initial batch of recent test runs when the page loads, or to fetch filtered runs when a user selects a project, release, sprint, or enters a search query.\n   * `GET /api/stats`: Used to fetch aggregated metrics (status distribution, run counts, project-wise pass rates, and sprint trends) which populate the counters and analytics charts.\n3. **Asset Loading (`http://localhost:9000/siphon-screenshots/...`):** When a test case fails and contains a failure screenshot link, the UI directly fetches and renders the physical PNG image from the public MinIO bucket.\n\n---\n\n## Repository Structure\n\nThe project is managed inside a Go workspace (`go.work` multi-module layout) alongside a React web application:\n\n```\nsiphon/\n├── apps/\n│   └── siphon-glass/            # Vite + React + TS dashboard client (Recharts, Lucide)\n├── services/\n│   ├── siphon-ingress/          # Go gRPC stream ingestion edge service (Binary Protobuf)\n│   ├── siphon-sink/             # Go consumer worker pool inserting to MongoDB\n│   └── siphon-stream-api/       # Go Gorilla WebSocket server tailing Mongo\n├── shared/\n│   ├── proto/                   # Protobuf schema and compiled Go bindings\n│   └── telemetry/               # Shared OpenTelemetry configuration utilities\n├── clients/\n│   ├── go-hook/                 # Mock client framework simulation\n│   └── seeder/                  # High-volume mock database seeder (1000s of records)\n├── docker/                      # MongoDB init indexing script\n└── docker-compose.yml           # Core database, broker, and LGPL observability configuration\n```\n\n---\n\n## Tech Stack \u0026 Port Configuration\n\n| Service | Port | Technology | Purpose |\n| :--- | :--- | :--- | :--- |\n| **`siphon-glass`** | `5173` | React, Recharts, Lucide | Premium, dark-mode real-time visual dashboard |\n| **`siphon-ingress`** | `50051` | Go, gRPC, RabbitMQ | High-performance metrics edge stream ingestion |\n| **`siphon-sink`** | *Daemon* | Go, MongoDB | Concurrent worker pool executing idempotent data writes |\n| **`siphon-stream-api`** | `8080` | Go, Gin, WebSockets | WebSockets and REST endpoint data stream tailer |\n| **MongoDB** | `27017` | MongoDB Community | Document database storing flat test metadata |\n| **Mongo Express** | `8081` | Web Console UI | Visual interface for exploring Mongo database collections |\n| **RabbitMQ** | `5672` / `15672` | RabbitMQ + Management | Event queue buffering ingestion spikes |\n| **MinIO** | `9000` / `9001` | MinIO Storage Engine | Object storage bucket storing failed test screenshots |\n| **Grafana** | `3000` | Grafana Dashboard | LGPL telemetry dashboard for metrics, traces, and Loki logs |\n\n---\n\n## Production-Grade Enhancements\n\nThe codebase implements several advanced systems engineering principles:\n* **Binary Protobuf Broker Pipeline**: Instead of slow JSON parsing on the edge API, `siphon-ingress` forwards raw binary protobuf packets (`proto.Marshal`) directly to RabbitMQ using the `application/x-protobuf` content-type. The consumer pool (`siphon-sink`) performs the deserialization, minimizing ingestion latency.\n* **Dead Letter Exchanges (DLX)**: If messages are malformed or fail to process, they are rejected with a `Nack(false, false)` and routed to the direct exchange `siphon-dlx-exchange` and stored in `siphon-dlq` to prevent consumer lockups.\n* **MongoDB Time-Series Storage**: The test results are persisted in a native MongoDB Time-Series collection partitioned chronologically via `timeField: \"timestamp\"` and `metaField: \"project\"`, ensuring query optimizations and data compaction.\n* **Idempotency Standard**: Unique compound index `{ \"execution_id\": 1, \"test_case_id\": 1 }` avoids duplicate runs.\n\n---\n\n## Setup \u0026 Running the Project\n\n### Prerequisites\n* Go v1.22+\n* Node.js v18+\n* Docker Desktop\n\n### 1. Launch local Infrastructure Services\n```bash\ndocker compose up -d\n```\n\n### 2. Seed the Database\nTo populate the dashboard with 1,250 historical test results across multiple projects (`siphon-core`, `payment-gateway`, `auth-service`, `notifications-engine`), releases, and sprints:\n```bash\ngo run clients/seeder/seeder.go\n```\n\n### 3. Run Go microservices (Concurrently in background)\n```bash\ngo run services/siphon-ingress/main.go\ngo run services/siphon-sink/main.go\ngo run services/siphon-stream-api/main.go\n```\n\n### 4. Start the Web Dashboard Client\n```bash\ncd apps/siphon-glass\nnpm run dev\n```\nOpen `http://localhost:5173/` in your browser. Use the top filters and search bar to drill down into the live test metrics.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpranavkakde%2Fsiphon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpranavkakde%2Fsiphon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpranavkakde%2Fsiphon/lists"}