{"id":27248845,"url":"https://github.com/bakdata/timely-autoscaling-demo","last_synced_at":"2026-07-01T17:31:36.598Z","repository":{"id":166464793,"uuid":"609028491","full_name":"bakdata/timely-autoscaling-demo","owner":"bakdata","description":"Kafka Summit 2023 - Timely Autoscaling of Complex Kafka Streams Pipelines","archived":false,"fork":false,"pushed_at":"2023-04-14T11:46:43.000Z","size":570,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-10T23:48:17.339Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bakdata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-03-03T08:19:01.000Z","updated_at":"2023-05-17T13:22:01.000Z","dependencies_parsed_at":null,"dependency_job_id":"2a943510-ab05-409c-9055-32ee637e5818","html_url":"https://github.com/bakdata/timely-autoscaling-demo","commit_stats":null,"previous_names":["bakdata/timely-autoscaling-demo"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/bakdata/timely-autoscaling-demo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Ftimely-autoscaling-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Ftimely-autoscaling-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Ftimely-autoscaling-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Ftimely-autoscaling-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bakdata","download_url":"https://codeload.github.com/bakdata/timely-autoscaling-demo/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Ftimely-autoscaling-demo/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35017089,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-01T02:00:05.325Z","response_time":130,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-10T23:48:16.508Z","updated_at":"2026-07-01T17:31:36.589Z","avatar_url":"https://github.com/bakdata.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Timely Auto-Scaling of Kafka Streams Pipelines with Remotely Connected APIs\n\n## Abstract\n\nCalling APIs over the network from Kafka Streams is often a necessary evil:\nAlthough it can incur significant costs by blocking the processing and thus making our pipelines less reliable,\nwe are sometimes forced to integrate business process-related microservices or\nmachine learning models and other technology that does not pair well with the JVM.\nWhile we scale the Kafka Streams applications and the services in Kubernetes to avoid blocked pipelines,\nit is often too late, as the scaling traditionally relies on metrics like the consumer group lag or number of HTTP requests.\n\nIn this talk, we first give an overview of the caveats when integrating such services in Kafka Streams and basic approaches for mitigating\nthose.\nSecond, we present our solution for the timely scaling of complex Kafka Streams pipelines in conjunction with remotely connected APIs.\nIn addition to the well-known metrics, we take further dimensions into account.\nPowered by Kafka Streams, we observe our data stream, extract metadata, aggregate statistics, and finally expose them as external metrics.\nWe integrate this with auto-scaling frameworks such as KEDA to reliably scale our pipelines just in time.\n\n## Setup\n\n## Pipelines\n\n### Basic Pipeline\n\n![Basic pipeline](docs/images/basic.png)\n\n---\n\n### Cached Pipeline\n\n![Cached pipeline](docs/images/cached.png)\n\n---\n\n### Pipeline with Auto-Scaling\n\n![Cached pipeline](docs/images/autoscaled.png)\n\n---\n\n### Pipeline with Timely Auto-Scaling\n\n![Cached pipeline](docs/images/timely.png)\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbakdata%2Ftimely-autoscaling-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbakdata%2Ftimely-autoscaling-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbakdata%2Ftimely-autoscaling-demo/lists"}