{"id":48865510,"url":"https://github.com/killme2008/greptimedb-ingestion-benchmark","last_synced_at":"2026-04-15T18:05:58.641Z","repository":{"id":345938261,"uuid":"1187017794","full_name":"killme2008/greptimedb-ingestion-benchmark","owner":"killme2008","description":"A CLI benchmark tool that compares GreptimeDB's ingestion performance across different protocols under identical conditions.","archived":false,"fork":false,"pushed_at":"2026-04-15T13:31:45.000Z","size":87,"stargazers_count":0,"open_issues_count":0,"forks_count":2,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-15T15:25:30.948Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/killme2008.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-20T08:46:35.000Z","updated_at":"2026-04-15T13:32:54.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/killme2008/greptimedb-ingestion-benchmark","commit_stats":null,"previous_names":["killme2008/greptimedb-ingestion-benchmark"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/killme2008/greptimedb-ingestion-benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/killme2008%2Fgreptimedb-ingestion-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/killme2008%2Fgreptimedb-ingestion-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/killme2008%2Fgreptimedb-ingestion-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/killme2008%2Fgreptimedb-ingestion-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/killme2008","download_url":"https://codeload.github.com/killme2008/greptimedb-ingestion-benchmark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/killme2008%2Fgreptimedb-ingestion-benchmark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31853304,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-15T15:24:51.572Z","status":"ssl_error","status_checked_at":"2026-04-15T15:24:39.138Z","response_time":63,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-15T18:05:57.643Z","updated_at":"2026-04-15T18:05:58.633Z","avatar_url":"https://github.com/killme2008.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GreptimeDB Ingestion Protocol Benchmark\n\nA CLI benchmark tool that compares GreptimeDB's ingestion performance across different protocols under identical conditions.\n\n## Protocols\n\n| Protocol | Go Client | Port | Transport |\n|----------|-----------|------|-----------|\n| gRPC SDK | `greptimedb-ingester-go` v0.7.1 | 4001 | gRPC |\n| gRPC Stream | `greptimedb-ingester-go` v0.7.1 | 4001 | gRPC (bidirectional streaming) |\n| gRPC Bulk (Arrow) | `greptimedb-ingester-go` v0.7.1 | 4001 | Arrow Flight (DoPut) |\n| InfluxDB Line Protocol | `influxdb-client-go` v2.14.0 | 4000 | HTTP |\n| OpenTelemetry (OTLP) | `go.opentelemetry.io/otel` v1.42.0 | 4000 | HTTP+Protobuf (Logs) |\n| MySQL INSERT | `go-sql-driver/mysql` v1.9.3 | 4002 | MySQL wire |\n| PostgreSQL INSERT | `jackc/pgx` v5.8.0 | 4003 | PostgreSQL wire |\n\nAll clients use default SDK settings — no connection pool tuning, no buffer size adjustments, no custom timeouts.\n\n## Quick Start\n\nThe one-liner script installs GreptimeDB (if needed), starts it, runs the benchmark, and cleans up on exit:\n\n```bash\nbin/run.sh\n```\n\nPass any flags through to the benchmark:\n\n```bash\nbin/run.sh -protocols grpc,influxdb -total-rows 50000 -concurrency 8\nbin/run.sh -protocols grpc,otel -batch-size 50,100,500,1000\nbin/run.sh --help\n```\n\nWhen `-host` is specified (via flag or `GREPTIME_HOST` env var), the script skips installing and starting a local GreptimeDB — you manage the instance yourself:\n\n```bash\nbin/run.sh -host 10.0.0.1 -protocols grpc_bulk\n```\n\n## Manual Usage\n\nIf you already have a GreptimeDB instance running:\n\n```bash\ngo build -o bin/ingestion-benchmark .\nbin/ingestion-benchmark -host 127.0.0.1 -total-rows 100000\n```\n\n## Configuration\n\nConnection settings can be configured via environment variables or CLI flags. CLI flags take precedence.\n\n| Environment Variable | CLI Flag | Default | Description |\n|---------------------|----------|---------|-------------|\n| `GREPTIME_HOST` | `-host` | `127.0.0.1` | GreptimeDB host |\n| `GREPTIME_DATABASE` | `-database` | `public` | Database name |\n| `GREPTIME_USER` | `-user` | (empty) | Username |\n| `GREPTIME_PASSWORD` | `-password` | (empty) | Password |\n\nYou can also put these in a `.env` file at the project root — `bin/run.sh` will pick them up.\n\n```bash\n# .env\nGREPTIME_HOST=10.0.0.1\nGREPTIME_DATABASE=benchmark\nGREPTIME_USER=admin\nGREPTIME_PASSWORD=secret\n```\n\n### Benchmark Flags\n\n```\n-protocols         Comma-separated: grpc,grpc_stream,grpc_bulk,influxdb,otel,mysql,postgres (default: all)\n-total-rows        Total rows to write per protocol (default: 10000000)\n-batch-size        Rows per batch, comma-separated for multiple (default: 1000)\n-concurrency       Number of concurrent workers (default: 5)\n-warmup-batches    Warm-up batches before measurement (default: 10)\n-num-hosts         Number of unique hosts; series = num-hosts × 1000 (default: 100 → 100K series)\n-seed              Random seed for data generation (default: 42)\n-output            Output format: table, json (default: table)\n```\n\n## Data Model\n\nEach protocol writes to its own isolated table to ensure fair comparison:\n\n| Protocol | Table |\n|----------|-------|\n| gRPC SDK | `benchmark_grpc` |\n| gRPC Stream | `benchmark_grpc_stream` |\n| gRPC Bulk (Arrow) | `benchmark_grpc_bulk` |\n| InfluxDB LP | `benchmark_influxdb` |\n| OpenTelemetry | `benchmark_otel` |\n| MySQL INSERT | `benchmark_mysql` |\n| PostgreSQL INSERT | `benchmark_postgres` |\n\nSchema:\n\n- **Tags**: `host` (configurable via `-num-hosts`), `region` (5), `datacenter` (10), `service` (20) — series = num-hosts × 1,000 (default 100,000)\n- **Fields**: `cpu`, `memory`, `disk_util`, `net_in`, `net_out` (all float64)\n- **Timestamp**: millisecond precision\n\nData is pre-generated with a fixed random seed before measurement begins, ensuring deterministic and reproducible results across runs.\n\n## How It Works\n\n1. **Generate** — All data points are pre-generated in memory (not timed). Warmup data and measurement data use separate, non-overlapping batches.\n2. **Setup** — Initialize the writer (connect, create per-protocol tables)\n3. **Warm-up** — Write warmup-only batches, results discarded\n4. **Measure** — N concurrent workers pull measurement batches from a shared channel; each `WriteBatch` call is timed. Streaming protocols (gRPC Stream, gRPC Bulk) use per-worker streams for maximum throughput.\n5. **Verify** — Query actual row count via HTTP SQL API and compare against expected; log warning on mismatch.\n6. **Report** — Compute throughput (rows/sec = total rows / wall clock time) and latency percentiles (p50/p95/p99)\n\nEach protocol is benchmarked sequentially. Within a protocol run, batches are distributed across workers via a channel-based work queue.\n\n## Verification\n\nAfter a run, the tool automatically verifies row counts. You can also check manually:\n\n```bash\nmysql -h 127.0.0.1 -P 4002 -e \"SHOW TABLES\" public\nmysql -h 127.0.0.1 -P 4002 -e \"SELECT COUNT(*) FROM benchmark_grpc\" public\n```\n\n## Requirements\n\n- Go 1.25+\n- `curl` and `nc` (for `bin/run.sh`)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkillme2008%2Fgreptimedb-ingestion-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkillme2008%2Fgreptimedb-ingestion-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkillme2008%2Fgreptimedb-ingestion-benchmark/lists"}