{"id":20252627,"url":"https://github.com/streamdal/kng","last_synced_at":"2025-04-10T23:21:22.976Z","repository":{"id":37947911,"uuid":"355375935","full_name":"streamdal/kng","owner":"streamdal","description":"High throughput oriented kafka wrapper lib","archived":false,"fork":false,"pushed_at":"2022-12-28T22:44:08.000Z","size":5613,"stargazers_count":5,"open_issues_count":0,"forks_count":2,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-03-24T20:11:12.500Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/streamdal.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-04-07T01:24:26.000Z","updated_at":"2022-12-20T09:01:50.000Z","dependencies_parsed_at":"2023-01-31T07:31:17.788Z","dependency_job_id":null,"html_url":"https://github.com/streamdal/kng","commit_stats":null,"previous_names":["batchcorp/kng"],"tags_count":24,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamdal%2Fkng","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamdal%2Fkng/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamdal%2Fkng/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/streamdal%2Fkng/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/streamdal","download_url":"https://codeload.github.com/streamdal/kng/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248313095,"owners_count":21082804,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T10:17:40.230Z","updated_at":"2025-04-10T23:21:22.955Z","avatar_url":"https://github.com/streamdal.png","language":"Go","readme":"kng\n===\n\n`kng` is a kafka client wrapper lib for Go that is geared towards facilitating\nhigh throughput message production and consumption.\n\nIt wraps both [segmentio/kafka-go](https://github.com/segmentio/kafka-go) and\n[Shopify/sarama](https://github.com/Shopify/sarama) libs for different\nfunctionality.\n\n## Special bits\n\nThe publisher does a lot of stuff under the hood which dramatically improves\nwrite performance.\n\n* Automatically creates a dedicated publisher for the given topic IF a \npublisher does not already exist.\n* Starts a background publisher in a goroutine that will clear its queue on\nan interval defined by `PublishInterval`.\n* Publisher goroutine will be stopped if it is idle for longer than \n`WorkerIdleTimeout`.\n* To avoid kafka from rejecting a batch containing too many messages, the\npublisher will automatically divide the batch into \"sub-batches\" (whose\nsize is defined by `DefaultSubBatchSize`.\n\n## Why multiple libs?\n\nThe segment lib works great for consuming and producing (and has internal\nbatching mechanisms) but doesn't work as well when dealing with topic\nmanagement and other administrative tasks.\n\nWe also prefer the `kafka-go` interface but some things just work better with\nthe `sarama` lib.\n\n## Name\n\n`kng` is \"kafka next generation\" which is _mildly_ poking fun at the late 90's\nsoftware naming pattern that included \"next generation\" in its name (such as\n`syslog-ng`) :smiley:\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstreamdal%2Fkng","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstreamdal%2Fkng","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstreamdal%2Fkng/lists"}