{"id":19109233,"url":"https://github.com/thearyanahmed/logflow","last_synced_at":"2026-05-09T06:08:34.530Z","repository":{"id":128883450,"uuid":"377443771","full_name":"thearyanahmed/logflow","owner":"thearyanahmed","description":"Proof of concept | Collecting nginx log data and pushing it to kafka with the help of gRPC.","archived":false,"fork":false,"pushed_at":"2021-06-26T18:40:29.000Z","size":2131,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-20T07:11:14.204Z","etag":null,"topics":["golang","grpc","kafka","nginx"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thearyanahmed.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-16T09:35:46.000Z","updated_at":"2024-12-25T11:56:04.000Z","dependencies_parsed_at":null,"dependency_job_id":"bb20287a-521a-4400-8e66-7f6fb8251d21","html_url":"https://github.com/thearyanahmed/logflow","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/thearyanahmed/logflow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thearyanahmed%2Flogflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thearyanahmed%2Flogflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thearyanahmed%2Flogflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thearyanahmed%2Flogflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thearyanahmed","download_url":"https://codeload.github.com/thearyanahmed/logflow/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thearyanahmed%2Flogflow/sbom","scorecard":{"id":877847,"data":{"date":"2025-08-11","repo":{"name":"github.com/thearyanahmed/logflow","commit":"2c623e19c0c23e603099163544225ed6eac98d02"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.2,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":0,"reason":"license file not detected","details":["Warn: project does not have a license file"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: containerImage not pinned by hash: docker/Dockerfile:1: pin your Docker image by updating golang:latest to golang:latest@sha256:9e56f0d0f043a68bb8c47c819e47dc29f6e8f5129b8885bed9d43f058f7f3ed6","Info:   0 out of   1 containerImage dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":8,"reason":"2 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GO-2023-2153 / GHSA-m425-mq94-257g / GHSA-qppj-fm5r-hxr3","Warn: Project is vulnerable to: GO-2024-2611 / GHSA-8r3f-844c-mc37"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-24T06:45:32.326Z","repository_id":128883450,"created_at":"2025-08-24T06:45:32.326Z","updated_at":"2025-08-24T06:45:32.326Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32809149,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-08T08:22:46.396Z","status":"online","status_checked_at":"2026-05-09T02:00:06.633Z","response_time":123,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","grpc","kafka","nginx"],"created_at":"2024-11-09T04:19:33.356Z","updated_at":"2026-05-09T06:08:34.509Z","avatar_url":"https://github.com/thearyanahmed.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Logflow \n\n### What it does?\nLogflow collects nginx layer 7 data and layer 4 packet data and will be sent to kafka using protobuf.\nAs of using protocol buffer, it works very fast. We will be improving on it. It is still in very early stage.\n\nIt takes the data from the gRPC request and uses publishes into kafka . Logflow uses `client stream` to collect the data.\n\nIt can publish a single request into multiple topics and topics can be changed on every request, as the time of writing this,\nall possible topics must be given (set in the `.env` file) before you start the server. \n\n### Dependencies\n1. [Kafka](https://kafka.apache.org/downloads)\n2. GO\n3. [Protocol Buffer](https://grpc.io/docs/protoc-installation/) for development purpose.\n\n*At the moment we don't have the docker environments ready yet. The dependencies need to be installed manually.*\n\n## Running the Application\n\nFirst, git clone\n\n```bash\n// https \ngit clone https://github.com/thearyanahmed/logflow.git\n\n// or ssh\ngit clone git@github.com:thearyanahmed/logflow.git\n\n// github cli\ngh repo clone thearyanahmed/logflow\n```\nthen cd into the directory\n\n```bash\ncd logflow\n```\n\nIf you don't have ZooKeeper and Kafka running,\n[you can follow these instructions](https://github.com/thearyanahmed/logflow#start-zookeeper-and-kafka \"Starting kafka\")\n\n\n\n### .ENV\nOnce you have ZooKeeper and Kafka running, in the `logflow` directory\n\n```bash\ncp .env.example to .env\n```\nMake sure you have the .env values setup correctly, in case you have changed any config for kafka or \nif some default ports/values are already in use on your machine.\n\nTo start the server, run \n\n```bash\ngo run main.go --action serve\n```\n\nThis is start tcp connect server at `RPC_PORT` ( from .env) . The default is 5053.\n\nThen you need to start the upd server by running\n\n```bash\ngo run server/udp_server.go\n```\n\nAdd the following to your nginx config,\n```text\nlog_format tufin escape=json\n'{'\n     '\"time\":\"$msec\",'\n     '\"connection\":\"$connection\",'\n     '\"request\":\"$request\",'\n     '\"status\":\"$status\",'\n     '\"user_agent\":\"$http_user_agent\"'\n'}';\n\n\n# and inside your server block \nserver {\n    ...\n    access_log syslog:server=localhost:6060,facility=local7,tag=nginx,severity=info tufin;\n}\n```\n\nMake sure you keep the `server=$host:$port` same as .env's `UDP_SERVER_PROT`\n\nAfter that, visit your web app that is using the nginx config,\nvisit any page, and it should start streaming access logs to kafka.\n\n**Notes** \n- Before running the client, make sure you have **ZooKeeper** and **Kafka** running, and the **topics** have been created.\n- Current docker image does not work, for what I believe due to a bug in [wurstmeister/kafka-docker#issue-516](https://github.com/wurstmeister/kafka-docker/issues/516), I get the same error as [segmentio/kafka-go#issues/682](https://github.com/segmentio/kafka-go/issues/682). Thus, you'll need to run kafka and zookeeper manually for the time being.\n- The program is at a very early stage. Program structure is subject to change.\n\n### Architecture\n![Logflow Architecture](images/logflow-architecture.png?raw=true \"Logflow Architecture\")\n\n\n#### Start ZooKeeper and Kafka\n\nFirst run ZooKeeper and Kafka brokers. Go to your kafka installation directory and run\n\n```\nbin/zookeeper-server-start.sh config/zookeeper.properties\n```\n\n```\nbin/kafka-server-start.sh config/server.properties\n```\n\nIf this is your first time running them, you probably don't have topics. You'll need at least one.\nTo create one, use the following command.\n\nIt will create a topic name `hello_world`\n```bash\nbin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic hello_world\n```\n\n- Running kafka consumer (optional)\n  In case you want to see the data transmission directly from kafka.\n\n```bash\nbin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic hello_world --from-beginning```\n```\n\n### With Docker\n\nAt the moment I have not docker image ready for this but soon will.\nIf you want to run kafka inside docker, you can simply use your docker\ncontainer's address eg: localhost:39092 or something like that ( in the .env) .\n\n### Flags\n\nThere is at the moment only one flag you can pass along when running the main.go file.\n\n1.  `--action` Possible values: serve, client, help. eg: `go run main.go --action serve`\n\n\n### Useful Resources\n\n[Enterprise Network Flow Collector (IPFIX, sFlow, Netflow) from Verizon Media](https://github.com/VerizonDigital/vflow)\n\n[The high-scalability sFlow/NetFlow/IPFIX collector used internally at Cloudflare](https://github.com/cloudflare/goflow)\n\n[GopherCon 2016: John Leon - Packet Capture, Analysis, and Injection with Go](https://www.youtube.com/watch?v=APDnbmTKjgM)\n\n[Capturing HTTP packets the hard way](https://medium.com/@cjoudrey/capturing-http-packets-the-hard-way-b9c799bfb6)\n\n[LISA16: Linux 4.X Tracing Tools: Using BPF Superpowers](https://www.youtube.com/watch?v=UmOU3I36T2U)\n\n[Sniffing Creds with Go, A Journey with libpcap](https://itnext.io/sniffing-creds-with-go-a-journey-with-libpcap-73bc3e74966)\n\n[Collecting NGINX Plus Monitoring Statistics with Go\n](https://www.nginx.com/blog/collecting-nginx-plus-monitoring-statistics-with-go/)\n\n[GoPacket by Google](https://pkg.go.dev/github.com/google/gopacket)\n\n[Packet Capture, Injection, and Analysis with Go Packet](https://www.devdungeon.com/content/packet-capture-injection-and-analysis-gopacket)\n\n[BCC HTTP Filter](https://github.com/iovisor/bcc/tree/master/examples/networking/http_filter)\n\n[Logkit by Qiniu](https://github.com/qiniu/logkit)\n\n[NGINX log data format](http://nginx.org/en/docs/http/ngx_http_log_module.html#log_formata)\n\nThese ^ are gems\n\n## Todos \n- [x] Installations \u0026 setup\n    - [x] Setup kafka\n    - [x] Use docker container for kafka\n    - [x] Decide on protobuf ( grpc / rpc )\n    - [x] Setup a nginx docker\n- [x] Enable .env support     \n- [x] Draw system architecture  \n- [x] Connect to kafka\n- [x] Setup a kafka client to receive messages\n- [x] Setup a kafka producer \n- [ ] Write tests\n- [x] Get some log data and test the whole system\n- [x] Dockerize full app\n- [x] Prepare dummy NginxLogRequest, Packet \u0026 Headers\n- [x] Prepare a client to test grpc and kafka producer with dummy data\n- [x] Also see manually if kafka consumer is consuming messages\n- [ ] Document everything ( on going so far)\n\n#### At the moment, I did not include any docker image for any component.\n\n#### More to come\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthearyanahmed%2Flogflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthearyanahmed%2Flogflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthearyanahmed%2Flogflow/lists"}