{"id":20961355,"url":"https://github.com/perbu/pkafka","last_synced_at":"2026-04-20T21:37:27.504Z","repository":{"id":143067726,"uuid":"609460061","full_name":"perbu/pkafka","owner":"perbu","description":null,"archived":false,"fork":false,"pushed_at":"2023-03-04T09:26:48.000Z","size":16,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-12-31T11:27:18.184Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/perbu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-04T08:33:59.000Z","updated_at":"2023-03-04T09:10:27.000Z","dependencies_parsed_at":null,"dependency_job_id":"f0a39bf2-9002-4684-9daf-628593318a2d","html_url":"https://github.com/perbu/pkafka","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/perbu/pkafka","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perbu%2Fpkafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perbu%2Fpkafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perbu%2Fpkafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perbu%2Fpkafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/perbu","download_url":"https://codeload.github.com/perbu/pkafka/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perbu%2Fpkafka/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32067622,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-20T21:26:33.338Z","status":"ssl_error","status_checked_at":"2026-04-20T21:26:22.081Z","response_time":94,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-19T02:03:28.372Z","updated_at":"2026-04-20T21:37:27.485Z","avatar_url":"https://github.com/perbu.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pkafka\n\nThis package implements a persistent writer for Kafka. It'll try to write to Kafka asynchronously, \nand if it fails, it'll retry until it succeeds. It'll persist the messages on either local or GCS \ndisk, and will retry from there if it fails.\nUpon initialization, it'll look for any messages that were not written to Kafka, and will try to\nwrite them again.\nIt's designed to be used in a long-running process, and it'll keep retrying until it succeeds.\n\nIt uses the amazing franz-go package to do the heavy lifting.\n\n## Usage\n\nYou initialize the write with the New() function. It takes a storage type, pkafka.LocalStorage \nor pkafka.GCSStorage, and a \"bucketname\". For GCS the bucketname is the name of the bucket, for local storage it is the \npath to the directory where the messages will be stores.\n\nIn addition, New takes a list of kgo.Option, which are passed to the franz-go client. It will add the option to enable \ntransactions, which are needed for the persistent writer to work. Without it, we'd possibly change the messaging order.\n\n\n## Example:\n\n```go\npackage main\n\nimport (\n\t\"context\"\n\t\"github.com/perbu/pkafka\"\n\t\"github.com/twmb/franz-go/pkg/kgo\"\n\t\"os\"\n\t\"os/signal\"\n\t\"sync\"\n)\n\nfunc main() {\n\t// Create a new writer\n\twriter, err := pkafka.New(pkafka.LocalStorage, \"my-dir\", kgo.SeedBrokers(\"localhost:9092\"))\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tdefer writer.Close()\n\tctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt)\n\tdefer cancel()\n\twg := \u0026sync.WaitGroup{}\n\twg.Add(1)\n\tgo func() {\n\t\tdefer wg.Done()\n\t\t// Start the writer\n\t\terr = writer.Run(ctx)\n\t\tif err != nil {\n\t\t\tpanic(err)\n\t\t}\n\t}()\n\t// create a message:\n\tmsg := kgo.Record{\n\t\tTopic: \"my-topic\",\n        Value: []byte(\"my-value\"),\n    }\n\t// Write a message\n\terr = writer.PersistentProduce(msg)\n    if err != nil {\n\t\tpanic(err)\n\t}\n    \n\t// Wait for the writer to finish (Ctrl+C)\n\twg.Wait()\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperbu%2Fpkafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fperbu%2Fpkafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperbu%2Fpkafka/lists"}