{"id":31822926,"url":"https://github.com/devfans/net-kafka-repeater","last_synced_at":"2025-10-11T14:20:37.525Z","repository":{"id":98959707,"uuid":"196338258","full_name":"devfans/net-kafka-repeater","owner":"devfans","description":"A kafka message repeater with data transferred via the Internet (TCP/UDP).","archived":false,"fork":false,"pushed_at":"2019-08-28T07:14:13.000Z","size":45,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-06-20T03:41:58.658Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/devfans.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-07-11T07:01:22.000Z","updated_at":"2019-08-28T07:05:45.000Z","dependencies_parsed_at":null,"dependency_job_id":"61f66738-7e4a-4330-a050-029699b300d2","html_url":"https://github.com/devfans/net-kafka-repeater","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/devfans/net-kafka-repeater","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devfans%2Fnet-kafka-repeater","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devfans%2Fnet-kafka-repeater/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devfans%2Fnet-kafka-repeater/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devfans%2Fnet-kafka-repeater/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/devfans","download_url":"https://codeload.github.com/devfans/net-kafka-repeater/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devfans%2Fnet-kafka-repeater/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279007496,"owners_count":26084313,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-11T02:00:06.511Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-11T14:20:29.173Z","updated_at":"2025-10-11T14:20:37.521Z","avatar_url":"https://github.com/devfans.png","language":"Go","readme":"# net-kafka-repeater\nA kafka message repeater with data transferred via the Internet (TCP/UDP).\n\n## Overview\n\nThe repeater is desgined to tranfer messages to a remote location and rewrite the message to the kafka message queue.\nThe sender would subscribe to a topic and when a message is fetched from the queue, it will send it out to the receiver.\nWhen the receiver get a message from sender it will deliver it to the local kafka message queue.\n\n## Data packet structure\n\nFLAG(2 bytes) + SIZE(4 bytes) + ID(4 bytes) + DATA\n\n`FLAG` specifies the message type: a authentication request or a valid data message\n\n`ID`   is a random `uint32` to identify the message for confirmation from receiver(Not implemented yet)\n\n\n## Security\n\nAfter a sender established a connection to the receiver it would send a login request with plain text password carried. If the password does not match the receiver end password, the connection would be dropped. Without a authentication success the sender can not send messages to the receiver.\n\nThe connection is not secured with encryption by default.\n\n## Reliability\n\nThe sender would maintain a outgoing connectin to the receiver, and when the connection is broken, it'll retry again later.\n\nIf `order` is true, the messages received on receiver end would be delivered to the kafka message queue one by one.\n\nA kafka message delivery failure would normally trigger a new delivery but no message loss is not guaranteed.\n\nHowever, the 'ack' flag is for this purpose(Not implemented yet). The idea is the sender needs to receive delivery success ack message for each kafka message before sending out the next message, but the performance and efficiency could be lower.\n\n## Networking\n\nTwo types of networking solutions are provided:\n\n+ Tcp: a valid tcp connection is proper for a stable network link\n\n+ Kcp: a udp based connection for fast reliable delivery on a unstable network link with extra traffic cost\n\n## Configuration\n\nSamples\n\nsender.json\n```\n{\n  \"mode\": \"sender\",\n  \"pidFile\": \"sender.pid\",\n  \"logFile\": \"sender.log\",\n  \"workDir\": \".\",\n  \"exe\": \"net-kafka-repeater\",\n  \"relay\": {\n    \"address\": \"0.0.0.0:2000\",\n    \"password\": \"Testing\",\n    \"ack\": true,\n    \"order\": true,\n    \"mode\": \"tcp\"\n  },\n  \"topic\": {\n    \"topic\": \"xxxxx\",\n    \"order\": true,\n    \"to_channel\": true,\n    \"params\": {\n      \"bootstrap.servers\": \"127.0.0.1:9092\",\n      \"broker.address.family\": \"v4\",\n      \"group.id\": \"test\",\n      \"session.timeout.ms\": 6000,\n      \"auto.offset.reset\": \"earliest\"\n    }\n  }\n}\n```\n\nreceiver.json\n```\n{\n  \"mode\": \"receiver\",\n  \"pidFile\": \"receiver.pid\",\n  \"logFile\": \"receiver.log\",\n  \"workDir\": \".\",\n  \"exe\": \"net-kafka-repeater\",\n  \"relay\": {\n    \"address\": \"0.0.0.0:2000\",\n    \"password\": \"xxxxxxxxxx\",\n    \"ack\": true,\n    \"mode\": \"tcp\"\n  },\n  \"topic\": {\n    \"topic\": \"xxxxx\",\n    \"order\": true,\n    \"to_channel\": true,\n    \"params\": {\n      \"bootstrap.servers\": \"127.0.0.1:9092\"\n    }\n  }\n}\n\n```\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevfans%2Fnet-kafka-repeater","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdevfans%2Fnet-kafka-repeater","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevfans%2Fnet-kafka-repeater/lists"}