{"id":20411627,"url":"https://github.com/muhlemmer/count","last_synced_at":"2026-05-08T04:31:29.145Z","repository":{"id":62864962,"uuid":"549663225","full_name":"muhlemmer/count","owner":"muhlemmer","description":"Request counting API for zitadel interview process.","archived":false,"fork":false,"pushed_at":"2022-10-23T22:30:36.000Z","size":165,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-05T03:18:33.999Z","etag":null,"topics":["cockroachdb","go","grpc","grpc-go","pgx","postgresql"],"latest_commit_sha":null,"homepage":"https://buf.build/muhlemmer/count","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/muhlemmer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-10-11T14:38:42.000Z","updated_at":"2024-08-04T15:34:58.000Z","dependencies_parsed_at":"2022-11-08T06:33:26.620Z","dependency_job_id":null,"html_url":"https://github.com/muhlemmer/count","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/muhlemmer/count","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muhlemmer%2Fcount","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muhlemmer%2Fcount/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muhlemmer%2Fcount/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muhlemmer%2Fcount/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/muhlemmer","download_url":"https://codeload.github.com/muhlemmer/count/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muhlemmer%2Fcount/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262151861,"owners_count":23266929,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cockroachdb","go","grpc","grpc-go","pgx","postgresql"],"created_at":"2024-11-15T05:53:30.844Z","updated_at":"2026-05-08T04:31:24.113Z","avatar_url":"https://github.com/muhlemmer.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Count\n\n[![Go](https://github.com/muhlemmer/count/actions/workflows/go.yml/badge.svg)](https://github.com/muhlemmer/count/actions/workflows/go.yml)\n[![codecov](https://codecov.io/gh/muhlemmer/count/branch/main/graph/badge.svg?token=QUSPX5SMBH)](https://codecov.io/gh/muhlemmer/count)\n[![Go Reference](https://pkg.go.dev/badge/github.com/muhlemmer/count.svg)](https://pkg.go.dev/github.com/muhlemmer/count)\n\nCount is a request counting API, build for the Zitadel interview process.\nIt provides endpoints for request counting of other services\nand retrieving the count metrics.\n\n## Features\n\n- PostgreSQL and CockroachDB support.\n  The latter is confirmed by running tests Github actions against a cockroachdb free cloud offering.\n- Counted \"requests\" are send over a streaming gRPC to the count API.\n- High level queueing and server middleware are provided.\n- Queues can be non-blocking and the API server uses concurrent jobs for database inserts.\n  So a server posting to this API will not suffer from performance issues, even on connection\n  failures to this API or between the API and database.\n\n## Usage\n\n### Input clients\n\nClients can dail a gRPC Client Connection to this API\nand use [pkg/queue](https://pkg.go.dev/github.com/muhlemmer/count/pkg/queue)\nto start sending request data:\n\n```\ncc, err := grpc.DialContext(context.TODO(), \"count.muhlemmer.com:443\",\n    grpc.WithTransportCredentials(insecure.NewCredentials()),\n    grpc.WithBlock(),\n)\nif err != nil {\n    panic(err)\n}\n\nq, err := NewCountAddClient(context.TODO(), cc)\nif err != nil {\n    panic(err)\n}\n\nq.QueueOrDrop(context.TODO(), \u0026countv1.AddRequest{\n    Method:           countv1.Method_GET,\n    Path:             \"/foo/bar\",\n    RequestTimestamp: timestamppb.Now(),\n})\n```\n\nHTTP servers can use [Middleware](https://pkg.go.dev/github.com/muhlemmer/count/pkg/queue#CountAddQueue.Middleware) instead:\n\n```\ns := \u0026http.Server{\n    Addr:    \":8080\",\n    Handler: q.Middleware(http.DefaultServeMux),\n}\ns.ListenAndServe()\n```\n\nOr [UnaryInterceptor](https://pkg.go.dev/github.com/muhlemmer/count/pkg/queue#CountAddQueue.UnaryInterceptor) for gRPC servers:\n\n```\ngrpc.NewServer(grpc.ChainUnaryInterceptor(\n    q.UnaryInterceptor(),\n))\n```\n\n### Retrieval clients\n\nClients which want to retrieve metrics can use gRPC.\nAPI documenation is available at https://buf.build/muhlemmer/count/docs/main:count.v1.\n\nIf this API where to be used in producion, I would consider moving to https://connect.build/ as gRPC and REST protocol, which for now has a too big impact.\n\n### Server\n\nEasiest way to run a count API server, is to use the\nDocker image, which is build automatically by the CI/CD.\nFirst, create a `.env` file:\n\n```\n# Driver name for the migrations.\n# Use `pgx` for postgresql and\n# `cockroachdb` for cockroachdb.\nMIGRATION_DRIVER=cockroachdb\n\n# Database connection URL, including secrets.\nDB_URL=postgresql://\u003cuser\u003e:\u003cpassword\u003e@\u003chost\u003e:\u003cport\u003e/\u003cdb\u003e?sslmode=verify-full\u0026options=--cluster%3D\u003cyour-cockroachdb-cluster\u003e\n```\n\nThen, start the server with Docker:\n\n```\ndocker run --env-file .env -p 7777:7777 ghcr.io/muhlemmer/count:main\n```\n\n## Architecture\n\nThe design goal of this project was to \"increase a counter when a API request\nis made\", in PostgreSQL or CockroachDB. However keeping a table with API\nmethod names and a incremental counter would not scale well. Requests can\nhappen concurrently and `UPDATE`s on the same row will need to be serialized by\nthe database. In the case of CockroachDB this would require distributed locks\nwith concensus through the raft log. Due to lock contention the count API would\nnot be able to keep up.\n\nInstead, for every request a timestamp is inserted in a table, along with a `method_id` which is a foreign key to a method and path table. `INSERT` does not require locking and can happen on the same table on multiple nodes concurrenly. CockroachDB takes care of replicating and merging the records in the background.\n\nDaily a cron job can call the\n[CountDailyTotals](https://buf.build/muhlemmer/count/docs/main:count.v1#count.v1.CountService.CountDailyTotals) endpoint.\nThis counts the requests by `method_id`, stores the result in a\n`daily_method_totals` table while deleting all rows for that day from the\n`requests` table. This keeps storage size pretty decent. Both `int` and `timestamptz` take 8 bytes, so 16 bytes per row. 1 milion request counts per day would result in just 16MB of storage by the end of each day.\n\nAs such it is \"cheap\" to read periodic reports from the `daily_method_totals` table , such as yealy, monthly or daily.\n\n## Lessons learned\n\nSome hickups in the process where encountered. As CockroachDB is supposed to be\nsomewhat compatible with PostgreSQL, I initally felt confident to go the\nCockroachDB way, even without previous experience. This cost me more time as\nexpected due to dirrent incompatibilties and suprises:\n\n1. The cockroachdb Docker image installation process is interactive, unlike the\n  PosgreSQL docker image which can be configured using environment variable\n  This makes cockroachDB unsuitable for devcontainers.\n2. cockroachdb doesn't have `pg_advisory_lock`, which is used by the migrations\n  tool I know. Instead, I had to learn to use `golang-migrate` which came with \n  its own pitfalls. Besides, due to the fact a seperate lock table must be used \n  by the tool, any failure would lead in stale locks and require manual \n  cleanups. (probably a bug the tool, I did not invesigate).\n3. Because of point 1, I opted to use a free serverless instance from cockroach \n  labs. Due to physical distance to the cloud location I experienced high \n  network latency \u003e100ms which made it a pain to `INSERT` test data. As a \n  second option I tried to use `COPY FROM`, which bugged out on me. https://\n  github.com/cockroachdb/cockroach/issues/52722\n4. CockroachDB uses 8 bytes for all integer types, where PostgreSQL uses 4 \n  bytes for `int`. When defining a `int` column and using a `pgtype.Int4` in \n  the code, errors would occur when connecting to CockroachDB later, due to \n  witdh mismatch. Even when the column is defined as `int`, cockroach still \n  sends `bigint` on the wire. Using just `pgtype.Int8` gave me trouble with \n  PostgreSQL. So in the end I defined all integer columns as `bigint` with \n  `pgtype.Int8` to solve the porability issue.\n\nIt was a bit of a learning curve and although I do see the positive case for\nproduction use of CockroachDB, it is still a bit of a pain as development DB.\nIn this case I opted to develop against a local PostgreSQL instance first and\ntest against CockroachDB later and in CI.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmuhlemmer%2Fcount","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmuhlemmer%2Fcount","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmuhlemmer%2Fcount/lists"}