{"id":13783886,"url":"https://github.com/lwolf/konsumerator","last_synced_at":"2026-03-12T03:31:32.993Z","repository":{"id":53055560,"uuid":"194389420","full_name":"lwolf/konsumerator","owner":"lwolf","description":"Kafka Consumer Operator. Kubernetes operator to manage consumers of unbalanced kafka topics with per-partition vertical autoscaling based on Prometheus metrics","archived":false,"fork":false,"pushed_at":"2023-02-25T01:32:58.000Z","size":11373,"stargazers_count":22,"open_issues_count":3,"forks_count":1,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-12-17T12:57:45.296Z","etag":null,"topics":["autoscaler","autoscaling","crd-controller","kafka","kafka-consumers","kubebuilder","kubernetes","kubernetes-controllers","kubernetes-operator"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lwolf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-06-29T09:44:07.000Z","updated_at":"2024-01-29T07:55:43.000Z","dependencies_parsed_at":"2024-01-18T16:00:10.224Z","dependency_job_id":"b3a6f3ef-b716-4932-ac2a-5c4f524db640","html_url":"https://github.com/lwolf/konsumerator","commit_stats":null,"previous_names":[],"tags_count":27,"template":false,"template_full_name":null,"purl":"pkg:github/lwolf/konsumerator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lwolf%2Fkonsumerator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lwolf%2Fkonsumerator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lwolf%2Fkonsumerator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lwolf%2Fkonsumerator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lwolf","download_url":"https://codeload.github.com/lwolf/konsumerator/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lwolf%2Fkonsumerator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30414276,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-12T00:40:14.898Z","status":"online","status_checked_at":"2026-03-12T02:00:07.260Z","response_time":114,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autoscaler","autoscaling","crd-controller","kafka","kafka-consumers","kubebuilder","kubernetes","kubernetes-controllers","kubernetes-operator"],"created_at":"2024-08-03T19:00:32.540Z","updated_at":"2026-03-12T03:31:32.976Z","avatar_url":"https://github.com/lwolf.png","language":"Go","funding_links":[],"categories":["Deployment"],"sub_categories":["Kubernetes"],"readme":"[![Docker Repository on Quay](https://quay.io/repository/lwolf/konsumerator/status \"Docker Repository on Quay\")](https://quay.io/repository/lwolf/konsumerator)\n[![Build Status](https://travis-ci.org/lwolf/konsumerator.svg?branch=master)](https://travis-ci.org/lwolf/konsumerator)\n[![Go Report Card](https://goreportcard.com/badge/github.com/lwolf/konsumerator)](https://goreportcard.com/report/github.com/lwolf/konsumerator)\n[![codecov](https://codecov.io/gh/lwolf/konsumerator/branch/master/graph/badge.svg)](https://codecov.io/gh/lwolf/konsumerator)\n\n# Konsumerator\n\n**Konsumerator** is a Kubernetes operator intended to automate management and resource allocations for \nkafka consumers. \n\n| Branch  | description |\n| ------------- | ------------- |\n| master (v1)  | Current stable version (only bugfix)  |\n| v2   | Semi-stable v2 branch (new features go there)  |\n\nOperator creates and manages `Consumer` CRD, for this it requires cluster-wide permissions.\n\n```yaml\napiVersion: konsumerator.lwolf.org/v1\nkind: Consumer\nmetadata:\n  name: consumer-sample\nspec:\n  numPartitions: 100\n  numPartitionsPerInstance: 1\n  name: \"test-consumer\"\n  namespace: \"default\"\n  autoscaler:\n    # only one provider is supported yet - prometheus.\n    # `prometheus` is configured using prometheus provider and user specific metrics\n    mode: \"prometheus\"\n    prometheus:\n      # minimum allowed period to query prometheus for the lag\n      # information to avoid DDoS of that service\n      minSyncPeriod: \"1m\"\n      # do not scale up if the lag is less than 5 minutes\n      tolerableLag: \"5m\"\n      # approximate consumption rate per CPU\n      ratePerCore: 20000\n      # approximate memory requirements per CPU\n      # if ratePerCore is 10k ops, this value is amount of\n      # RAM needed during the processing of this 10k ops\n      ramPerCore: \"100M\"\n      # criticalLag is some value close to the SLO.\n      # if lag has reached this point, autoscaler will\n      # give maximum allowed resource to that deployment\n      criticalLag: \"60m\"\n      # preferable recovery time. During lag, Consumer will try to allocate \n      # as much resources as possible to recover from lag during this period\n      recoveryTime: \"30m\"\n      # prometheus addresses to query\n      address:\n        - \"http://prometheus-operator-prometheus.monitoring.svc.cluster.local:9090\"\n      # Offset query should return number of messages that is not\n      # processed yet a.k.a lag per partitionLabel\n      offset:\n        query: \"sum(rate(kafka_messages_last_offset{topic=''}[5m])) by (partition)\"\n        partitionLabel: \"partition\"\n      # Production query should return number of messages is being\n      # produced per partitionLabel per unit of time (second)\n      production:\n        query: \"sum(rate(kafka_messages_produced_total{topic=''}[5m])) by (partition)\"\n        partitionLabel: \"partition\"\n      # Consumption query should return number of messages is being\n      # consumed per partitionLabel per unit of time (second)\n      consumption:\n        query: \"sum(rate(kafka_messages_consumed_total{topic=''}[5m])) by (partition)\"\n        partitionLabel: \"partition\"\n  # partitionEnvKey - the name of the environment variable\n  # containing partition number the deployment is responsible for\n  partitionEnvKey: \"PARTITION\"\n  # DeploymentSpec to run the consumer\n  deploymentTemplate:\n    replicas: 1\n    strategy:\n      type: Recreate\n    selector:\n      matchLabels:\n        app: my-dep\n    template:\n      metadata:\n        labels:\n          app: my-dep\n      spec:\n        containers:\n          - image: busybox\n            name: busybox-sidecar\n            command: [\"/bin/sh\", \"-ec\", \"env \u0026\u0026 sleep 3000\"]\n          - image: busybox\n            name: busybox\n            command: [\"/bin/sh\", \"-ec\", \"sleep 2000\"]\n  # resource boundaries, this policy protects\n  # the consumer from scaling to 0 or infinity in case\n  # of incidents\n  resourcePolicy:\n    containerPolicies:\n    - containerName: busybox\n      minAllowed:\n        cpu: \"100m\"\n        memory: \"100M\"\n      maxAllowed:\n        cpu: \"1\"\n        memory: \"1G\"\n    - containerName: busybox-sidecar\n      minAllowed:\n        cpu: \"100m\"\n        memory: \"100M\"\n      maxAllowed:\n        cpu: \"100m\"\n        memory: \"100M\"\n```\n\nWhen such Consumer are being created, operator will create `.spec.numPartitions` unique deployments.\n\n```bash\n$ kubectl get consumers\n\n  NAME              EXPECTED   RUNNING   PAUSED   MISSING   LAGGING   OUTDATED   AUTOSCALER   AGE\n  consumer-sample   100        100       0        0         6         0          prometheus   6d2h\n\n\n```\n\nDeployment specifics:\n\n* Deployment will be created based on the template provided in `.spec.deploymentTemplate` without any modifications.\nIt does not make sense to set resource field, since it will be overridden by autoscaler.\n* Deployment will be named `{consumerName}-{index}` where consumerName is `.spec.name` and index is in range\nfrom 0 to `.spec.numPartitions`. \n* Resource requests/limits for each deployment will be estimated based on metrics and configuration\n* Each container in the deployment will get few environment variables set:\n    `KONSUMERATOR_PARTITION` - contains comma-separated list of kafka partition numbers assigned to this deployment. Name of this variable is configurable. \n    `KONSUMERATOR_NUM_PARTITIONS` - total number of kafka partitions \n    `KONSUMERATOR_INSTANCE` - ordinal of the instance \n    `KONSUMERATOR_NUM_INSTANCES` - total number of instances \n    `GOMAXPROCS` - golang specific setting, always equals to the `resources.limit.cpu`. \n\n## Metrics Providers\n\nAt the moment only one metrics provider is implemented - Prometheus.\n\nHere is a critical settings for the provider:\n\n```yaml\nspec:\n  ...\n  autoscaler:\n    prometheus:\n      # minimum allowed period to query prometheus for the lag\n      # information to avoid DDoS of that service\n      minSyncPeriod: \"1m\"\n      address:\n        - \"http://prometheus-operator-prometheus.monitoring.svc.cluster.local:9090\"\n      # Offset query should return number of messages that is not\n      # processed yet a.k.a lag per partitionLabel\n      offset:\n        query: \"sum(rate(kafka_messages_last_offset{topic=''}[5m])) by (partition)\"\n        partitionLabel: \"partition\"\n      # Production query should return number of messages is being\n      # produced per partitionLabel per unit of time (second)\n      production:\n        query: \"sum(rate(kafka_messages_produced_total{topic=''}[5m])) by (partition)\"\n        partitionLabel: \"partition\"\n      # Consumption query should return number of messages is being\n      # consumed per partitionLabel per unit of time (second)\n      consumption:\n        query: \"sum(rate(kafka_messages_consumed_total{topic=''}[5m])) by (partition)\"\n        partitionLabel: \"partition\"\n```\n\n\n## Resource Predictors\n\nAt the moment only one resource predictor is implemented which is tightly coupled with Prometheus metrics\nprovider.\n`NaivePredictor` operates using following settings provided by a user in `.spec.autoscaler.prometheus`\n* `ratePerCore`- approximate number of message that could be processed by a single core during normal operations\n* `ramPerCore` - approximate amount of RAM required for the amount of messages process by a core. Sometimes you don't have\nsuch dependency between cpu and memory and want to allocate fixed amount of memory, in this case, put here any value and set resourcePolicy\nfor the container with minAllowed.memory = maxAllowed.memory.\n* `recoveryTime` - if there is a lag of the partition, predictor will try to give consumer\nas much resources as possible to recover during this period.\nIt also require `production` and `offset` metrics from the Prometheus. \n\n## Guest Mode (no cluster wide permissions)\n\nSometimes you don't have permissions to create CRDs in the cluster, for such cases Konsumerator supports\na so-called `guest-mode`. When running in guest-mode operator uses configmaps instead of CRDs and it is limited to a \nsingle namespace.\nGuest-mode could be activated by setting namespace argument `konsumerator --namespace=default`.\n\nTo create an instance of the consumer, you need to create a ConfigMap with `konsumerator.lwolf.org/managed` annotation\nand consumerSpec inside the body. \n\n```\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: consumer-sample\n  namespace: default\n  annotations:\n    konsumerator.lwolf.org/managed: \"true\"\ndata:\n  consumer.yaml: |\n    numPartitions: 100\n    name: \"test-consumer\"\n    namespace: \"default\"\n    autoscaler:\n      ...\n```\n\n## Installation\n\ncoming soon...\n\n## Operations\n\n### How to stop/start consumer\n\nAt the moment, the only way to stop the consumer is to set `.spec.deploymentTemplate.replicas` to 0.\nThis will trigger reconciliation, operator will notice that the deployment spec was changed and\napply the change, in this case change the number of replicas to 0. \n\n\n### How to pause auto scaling  \n\nIt is possible to temporary disable autoscaling of the managed deployments. \nTo do so, add the following annotation to the `consumer` object with any value:\n\n```yaml\nannotations:\n  konsumerator.lwolf.org/disable-autoscaler: \"true\"\n```\n\nIn case of disabled autoscaling, operator will allocate minimum resources from the `resourcePolicy` for this container.\nIf there is no such policy set, no resource request/limit will be set.\n\n\n## Development\n\nRequirements\n\n* https://github.com/kubernetes-sigs/kind - is used to spin-up dev cluster (v0.16+)\n* https://github.com/etcd-io/etcd - etcd needs to be present in the system path. Kubebuilder is a using it to run integration tests locally\n* https://github.com/kubernetes-sigs/kustomize - needs to be preset in the system path to render generated manifests \n\n### Dev k8s cluster\n\nTo spin up dev k8s cluster run the following:\n\n```\nmake kind-create\n```\n\nThis will create 2 node cluster (1 master and 1 node) with pre-installed Prometheus and Grafana.\n\n#### Access cluster resources (Linux):\nKIND is configured to expose 2 ports.\nTo get IP address of the KIND worker node, run:\n\n```\nkubectl get nodes konsumerator-worker -o jsonpath='{ $.status.addresses[?(@.type==\"InternalIP\")].address }'\n```\nthen, you can access grafana and prometheus on the following ports of that IP address: \n\n* 30666 - prometheus\n* 30777 - grafana (admin/admin)\n\n\n`make build`\n\n`make test`\n\n`make install`\n\n`make run`\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flwolf%2Fkonsumerator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flwolf%2Fkonsumerator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flwolf%2Fkonsumerator/lists"}