{"id":28379351,"url":"https://github.com/bitquery/stream_protobuf_example","last_synced_at":"2026-02-15T21:03:14.035Z","repository":{"id":276278296,"uuid":"928739238","full_name":"bitquery/stream_protobuf_example","owner":"bitquery","description":"Example of project reading protobuf stream from kafka","archived":false,"fork":false,"pushed_at":"2025-05-27T14:14:47.000Z","size":68,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-23T10:50:44.093Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bitquery.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-07T06:34:02.000Z","updated_at":"2025-05-27T14:14:52.000Z","dependencies_parsed_at":"2025-02-07T09:34:15.467Z","dependency_job_id":"d97bceae-bbc8-4964-bee2-48796ba63f1b","html_url":"https://github.com/bitquery/stream_protobuf_example","commit_stats":null,"previous_names":["bitquery/stream_protobuf_example"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/bitquery/stream_protobuf_example","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitquery%2Fstream_protobuf_example","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitquery%2Fstream_protobuf_example/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitquery%2Fstream_protobuf_example/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitquery%2Fstream_protobuf_example/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bitquery","download_url":"https://codeload.github.com/bitquery/stream_protobuf_example/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitquery%2Fstream_protobuf_example/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29489343,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-15T19:29:10.908Z","status":"ssl_error","status_checked_at":"2026-02-15T19:29:10.419Z","response_time":118,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-30T02:38:11.675Z","updated_at":"2026-02-15T21:03:14.029Z","avatar_url":"https://github.com/bitquery.png","language":"Go","readme":"# stream_protobuf_example\nExample of project reading protobuf stream from kafka\n\n## Run project\n\ncheckout:\n\n```\ngit clone https://github.com/bitquery/stream_protobuf_example.git\n```\n\ninstall golang (https://go.dev/doc/install) and compile:\n\nyou may need to install C++\n\n```\napt-get install g++\n```\n\nand then \n\n```\ncd stream_protobuf_example/\ngo build\n```\n\n\n\ncopy config:\n\n```\ncp config_example.yml config.yml\n```\n\nthen fill in variables ```\u003cYOUR USERNAME\u003e``` and ```\u003cYOUR PASSWORD\u003e``` in config.yml file\n\n\nYou also can change the desired topic name:\n\nProtobuf format, no block aggregation\n```\nsolana.dextrades.proto\nsolana.tokens.proto\nsolana.transactions.proto\n```\n\n\nrun it:\n\n```\n./stream_protobuf_example \n%4|1738918885.417|CONFWARN|rdkafka#producer-1| [thrd:app]: Configuration property group.id is a consumer property and will be ignored by this producer instance\n%4|1738918885.417|CONFWARN|rdkafka#producer-1| [thrd:app]: Configuration property enable.auto.commit is a consumer property and will be ignored by this producer instance\nSubscribing to topic solana.dextrades.proto 4 partitions: [{0 Success 3 [3] [3]} {1 Success 2 [2] [2]} {2 Success 2 [2] [2]} {3 Success 1 [1] [1]}]...\nAssigned 4 consumers to solana.dextrades.proto topic\npress Ctrl-C to exit\nRunning consumer rdkafka#consumer-2\nRunning consumer rdkafka#consumer-5\nRunning consumer rdkafka#consumer-3\nRunning consumer rdkafka#consumer-4\nslot 319062398 processed 2 txs (2 trades) from partition 3[894333] in worker 3\nslot 319062398 processed 1 txs (1 trades) from partition 3[894336] in worker 7\nslot 319062398 processed 4 txs (6 trades) from partition 3[894332] in worker 4\nslot 319062398 processed 5 txs (8 trades) from partition 3[894335] in worker 5\n...\n```\n\nafter pressing Ctrl-C it will show basic statistical report\n\n## Checking speed of message reading\n\nTo test how fast your server reads messages, try the following experiment:\n\n1. Set ```enable.auto.commit: true``` in config.yml and set ```group.id: \u003cYOUR USERNAME\u003e_\u003cSOME RANDOM NUMBER\u003e```\n2. Run consumer for some time, stop it. Wait a minute and start again\n\nWhat you expect to see is the reducing lag in messages, from approx 60,000 msec to close to 0.\n\nIt must catch up in less than 10 seconds. If it does not catch up fast, test server connection speed and ping to brokers.\n\n\n# Docker \n\n## Build\n```\ndocker build -t stream_protobuf_example:v0.0.1 .\n```\n\n## Run\n```\ndocker run --rm -v $(pwd)/config.yml:/app/config.yml stream_protobuf_example:v0.0.1\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbitquery%2Fstream_protobuf_example","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbitquery%2Fstream_protobuf_example","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbitquery%2Fstream_protobuf_example/lists"}