{"id":50791515,"url":"https://github.com/timur-developer/clickpulse","last_synced_at":"2026-06-12T11:30:40.240Z","repository":{"id":347611417,"uuid":"1193782154","full_name":"timur-developer/clickpulse","owner":"timur-developer","description":"ClickHouse event collector with batching, Prometheus metrics, and Grafana dashboards.","archived":false,"fork":false,"pushed_at":"2026-06-07T11:43:19.000Z","size":70,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-07T13:06:41.867Z","etag":null,"topics":["batch-processing","clickhouse","docker","event-ingestion","go","golang","grafana","http-api","kubernetes","observability","prometheus"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/timur-developer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-27T15:18:47.000Z","updated_at":"2026-06-07T11:43:23.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/timur-developer/clickpulse","commit_stats":null,"previous_names":["timur-developer/metrics-batch-collector","timur-developer/clickpulse"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/timur-developer/clickpulse","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timur-developer%2Fclickpulse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timur-developer%2Fclickpulse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timur-developer%2Fclickpulse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timur-developer%2Fclickpulse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/timur-developer","download_url":"https://codeload.github.com/timur-developer/clickpulse/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/timur-developer%2Fclickpulse/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34243051,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-12T02:00:06.859Z","response_time":109,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch-processing","clickhouse","docker","event-ingestion","go","golang","grafana","http-api","kubernetes","observability","prometheus"],"created_at":"2026-06-12T11:30:39.483Z","updated_at":"2026-06-12T11:30:40.234Z","avatar_url":"https://github.com/timur-developer.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# clickpulse — ClickHouse event collector with batching and observability\n\n![clickpulselogo](https://raw.githubusercontent.com/timur-developer/clickpulse/refs/heads/main/clickpulse_logo.png)\n\n![Go](https://img.shields.io/badge/go-1.22%2B-00ADD8?logo=go\u0026logoColor=white)\n![ClickHouse](https://img.shields.io/badge/ClickHouse-FFCC01?logo=clickhouse\u0026logoColor=black)\n![Prometheus](https://img.shields.io/badge/Prometheus-E6522C?logo=prometheus\u0026logoColor=white)\n![Grafana](https://img.shields.io/badge/Grafana-F46800?logo=grafana\u0026logoColor=white)\n![License MIT](https://img.shields.io/badge/license-MIT-blue.svg)\n\nRead this in other languages: [Russian](README.ru.md)\n\n`clickpulse` is a Go service for receiving analytical events over HTTP and writing them to ClickHouse in batches.\n\nIt buffers accepted events in memory and flushes them by batch size or by time interval, reducing insert overhead while keeping the ingestion API simple.\n\n`clickpulse` also provides observability for the ingestion and write pipeline: Prometheus metrics and Grafana dashboards help track incoming events, batch flushes, HTTP latency, current batch size, and ClickHouse insert errors.\n\n## Contents\n\n- [Why](#why)\n- [What It Does](#what-it-does)\n- [How It Works](#how-it-works)\n- [Quick Start](#quick-start)\n- [API](#api)\n  - [POST /events](#post-events)\n  - [GET /healthz](#get-healthz)\n  - [GET /metrics](#get-metrics)\n- [Configuration](#configuration)\n- [Observability](#observability)\n- [Docker Compose](#docker-compose)\n- [Kubernetes](#kubernetes)\n- [Development](#development)\n- [License](#license)\n\n## Why\n\nAnalytical events are often produced by backend services, landing pages, scripts, or internal tools. Writing each event to ClickHouse as a separate insert can add unnecessary database pressure and makes it harder to understand what is happening in the ingestion path.\n\n`clickpulse` focuses on the core flow of an event ingestion pipeline:\n\n- receive events through a simple HTTP API\n- validate incoming JSON before accepting it\n- collect accepted events in an in-memory batcher\n- flush events to ClickHouse by batch size or by time interval\n- expose observability for HTTP traffic, accepted events, batch state, flushes, and ClickHouse insert errors\n- run the supporting infrastructure locally with Docker Compose\n\n## What It Does\n\n| Area | Description |\n| --- | --- |\n| HTTP ingestion | Accepts analytical events through `POST /events` |\n| Validation | Checks required fields and JSON payload shape before accepting an event |\n| Batching | Buffers events in memory and flushes them by size or interval |\n| Storage | Writes accepted events to ClickHouse |\n| Health checks | Exposes `GET /healthz` for service health |\n| Metrics | Exposes Prometheus metrics for ingestion traffic, batching, and ClickHouse write errors |\n| Observability | Includes a Grafana dashboard for monitoring the ingestion and write pipeline |\n| Local stack | Provides Docker Compose setup for clickpulse, ClickHouse, Prometheus, and Grafana |\n| Deployment base | Provides Kubernetes manifests in `k8s/` |\n\n## How It Works\n\n```mermaid\nflowchart LR\n    Client[HTTP client] --\u003e|POST /events| API[Go HTTP service]\n    API --\u003e Validator[JSON validation]\n    Validator --\u003e Batcher[In-memory batcher]\n    Batcher --\u003e|flush by size| CH[(ClickHouse)]\n    Batcher --\u003e|flush by interval| CH\n    Prometheus[Prometheus] --\u003e|scrape /metrics| API\n    Grafana[Grafana] --\u003e Prometheus\n```\n\nThe service receives events through `POST /events`, validates the JSON payload, and puts accepted events into an in-memory batcher.\n\nThe batcher flushes events to ClickHouse when one of these conditions is met:\n\n- the batch reaches `BATCH_SIZE`\n- `FLUSH_INTERVAL` passes since the previous flush\n\nThis keeps the API simple while avoiding row-by-row inserts into ClickHouse.\n\n## Quick Start\n\nClone the repository and run the local stack:\n\n```bash\ngit clone https://github.com/timur-developer/clickpulse.git\ncd clickpulse\ndocker compose up --build -d\n```\n\nAfter startup, the services are available at:\n\n| Service | URL |\n| --- | --- |\n| clickpulse | `http://localhost:8080` |\n| ClickHouse HTTP | `http://localhost:8123` |\n| Prometheus | `http://localhost:9090` |\n| Grafana | `http://localhost:3000` |\n\nSend a test event:\n\n```bash\ncurl -X POST http://localhost:8080/events \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"event_type\": \"page_view\",\n    \"source\": \"landing\",\n    \"user_id\": \"u123\",\n    \"value\": 1,\n    \"created_at\": \"2026-03-27T12:00:00Z\"\n  }'\n```\n\nCheck the service health:\n\n```bash\ncurl http://localhost:8080/healthz\n```\n\nCheck Prometheus metrics:\n\n```bash\ncurl http://localhost:8080/metrics\n```\n\n## API\n\n### `POST /events`\n\nAccepts a single analytical event in JSON format.\n\nExample request:\n\n```json\n{\n  \"event_type\": \"page_view\",\n  \"source\": \"landing\",\n  \"user_id\": \"u123\",\n  \"value\": 1,\n  \"created_at\": \"2026-03-27T12:00:00Z\"\n}\n```\n\nFields:\n\n| Field | Type | Required | Description |\n| --- | --- | --- | --- |\n| `event_type` | string | yes | Event name, for example `page_view`, `signup`, `click` |\n| `source` | string | yes | Event source, for example `landing`, `api`, `mobile` |\n| `user_id` | string | no  | User or client identifier |\n| `value` | number | no  | Numeric event value |\n| `created_at` | string | yes | Event timestamp in RFC3339 format |\n\nSuccessful response:\n\n```json\n{\"status\":\"accepted\"}\n```\n\nInvalid payloads return `400 Bad Request` with an error message.\n\n### `GET /healthz`\n\nHealth check endpoint.\n\n```bash\ncurl http://localhost:8080/healthz\n```\n\nExample response:\n\n```json\n{\"status\":\"ok\"}\n```\n\n### `GET /metrics`\n\nPrometheus scraping endpoint.\n\n```bash\ncurl http://localhost:8080/metrics\n```\n\nThe endpoint exposes metrics for HTTP traffic, accepted events, batch state, batch flushes, and ClickHouse insert errors.\n\n## Configuration\n\nThe service is configured through environment variables.\n\n| Variable | Description | Example |\n| --- | --- | --- |\n| `HTTP_PORT` | HTTP server port | `8080` |\n| `CLICKHOUSE_DSN` | ClickHouse connection string | `http://localhost:8123?user=app\u0026password=app` |\n| `BATCH_SIZE` | Number of events that triggers a flush | `100` |\n| `FLUSH_INTERVAL` | Time interval that triggers a flush | `5s` |\n| `LOG_LEVEL` | Application log level | `info` |\n\nExample local configuration:\n\n```bash\nexport HTTP_PORT=8080\nexport CLICKHOUSE_DSN=\"http://localhost:8123?user=app\u0026password=app\"\nexport BATCH_SIZE=100\nexport FLUSH_INTERVAL=5s\nexport LOG_LEVEL=info\n```\n\n## Observability\n\n`clickpulse` exposes Prometheus metrics and includes a Grafana dashboard for observing the ingestion and write pipeline.\n\nUseful signals to watch:\n\n- request rate for `POST /events`\n- HTTP latency\n- number of accepted events\n- current batch size\n- batch flush count\n- ClickHouse insert errors\n\nThis makes it easier to answer questions like:\n\n- Is the service receiving events?\n- Are requests getting slower?\n- Are batches flushing regularly?\n- Are ClickHouse inserts failing?\n- Does changing `BATCH_SIZE` or `FLUSH_INTERVAL` affect throughput?\n\n## Docker Compose\n\nThe Docker Compose setup runs clickpulse together with ClickHouse, Prometheus, and Grafana.\n\nTypical workflow:\n\n```bash\ndocker compose up --build -d\ndocker compose ps\ndocker compose logs -f clickpulse\n```\n\nStop the stack:\n\n```bash\ndocker compose down\n```\n\nRemove volumes as well:\n\n```bash\ndocker compose down -v\n```\n\n## Kubernetes\n\nKubernetes manifests are stored in `k8s/`.\n\nApply them:\n\n```bash\nkubectl apply -f k8s/\n```\n\nThe manifests provide a base setup for running the service in a cluster. They expect ClickHouse to be available through `CLICKHOUSE_DSN`.\n\n## Development\n\nRun tests:\n\n```bash\ngo test ./...\n```\n\nRun locally without Docker Compose:\n\n```bash\nexport HTTP_PORT=8080\nexport CLICKHOUSE_DSN=\"http://localhost:8123?user=app\u0026password=app\"\nexport BATCH_SIZE=100\nexport FLUSH_INTERVAL=5s\n\ngo run ./cmd/app\n```\n\nBuild the service:\n\n```bash\ngo build ./...\n```\n\n## License\n\nMIT. See [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimur-developer%2Fclickpulse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftimur-developer%2Fclickpulse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftimur-developer%2Fclickpulse/lists"}