{"id":13581974,"url":"https://github.com/neptune-networks/flow-exporter","last_synced_at":"2025-04-06T10:33:10.523Z","repository":{"id":43327478,"uuid":"215887134","full_name":"neptune-networks/flow-exporter","owner":"neptune-networks","description":"Export network flows from Kafka to Prometheus","archived":false,"fork":false,"pushed_at":"2023-02-25T05:47:58.000Z","size":7355,"stargazers_count":125,"open_issues_count":9,"forks_count":20,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-11-05T21:45:32.869Z","etag":null,"topics":["ipfix","kafka","netflow","pmacct","prometheus-exporter","sflow"],"latest_commit_sha":null,"homepage":"https://brooks.sh/2019/11/17/network-flow-analysis-with-prometheus/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/neptune-networks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-10-17T21:17:13.000Z","updated_at":"2024-10-11T17:15:07.000Z","dependencies_parsed_at":"2024-01-17T05:25:10.633Z","dependency_job_id":"cf666ea0-c7d7-4f17-8595-f8acce19a2e5","html_url":"https://github.com/neptune-networks/flow-exporter","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neptune-networks%2Fflow-exporter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neptune-networks%2Fflow-exporter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neptune-networks%2Fflow-exporter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neptune-networks%2Fflow-exporter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/neptune-networks","download_url":"https://codeload.github.com/neptune-networks/flow-exporter/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247470564,"owners_count":20944146,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ipfix","kafka","netflow","pmacct","prometheus-exporter","sflow"],"created_at":"2024-08-01T15:02:21.596Z","updated_at":"2025-04-06T10:33:08.557Z","avatar_url":"https://github.com/neptune-networks.png","language":"Go","funding_links":[],"categories":["Go","others"],"sub_categories":[],"readme":"# Flow Exporter\n\nFlow exporter is a tool that can take flow data (Netflow, sFlow, IPFIX) from Kafka and export it to [Prometheus](https://prometheus.io). These flow records can be helpful to visualize which autonomous systems traffic is coming from and going to.\n\nGrafana is a great tool to visualize Prometheus data, and can be used to take the flow data and visualized as so:\n\n![preview](https://user-images.githubusercontent.com/934497/67167662-85e57080-f36a-11e9-96e2-f6f5b3b7e5d0.png)\n\nAn in depth guide on setting this up on a Linux-based router can be found [here](https://brooks.sh/2019/11/17/network-flow-analysis-with-prometheus/).\n\n## Usage\n\nThe exporter can be started with:\n\n```\n./flow-exporter --brokers=kafka.fqdn.com:9092 --topic=pmacct.acct --asn=15169\n```\n\n- `--addr`: Listening address (default = :9590)\n- `--brokers`: A comma separated list of Kafka brokers (with their corresponding ports) to consume flows from\n- `--topic`: The Kafka topic to consume flows from\n- `--asn`: The autonomous system number that the flows are being monitored from\n\nOnce running, you can view the data by visiting [http://localhost:9590/metrics](http://localhost:9590/metrics).\n\nAn example of the Prometheus metrics you can find are:\n\n```\n# HELP flow_receive_bytes_total Bytes received.\n# TYPE flow_receive_bytes_total counter\nflow_receive_bytes_total{destination_as=\"397143\",destination_as_name=\"NEPTUNE-NETWORKS - Neptune Networks\",hostname=\"border.neptunenetworks.org\",source_as=\"10318\",source_as_name=\"CABLEVISION S.A.\"} 663\n\n# HELP flow_transmit_bytes_total Bytes transferred.\n# TYPE flow_transmit_bytes_total counter\nflow_transmit_bytes_total{destination_as=\"10318\",destination_as_name=\"CABLEVISION S.A.\",hostname=\"border.neptunenetworks.org\",source_as=\"397143\",source_as_name=\"NEPTUNE-NETWORKS - Neptune Networks\"} 1137\n```\n\nFlow Exporter automatically finds the name of the ASN and adds it to the metric.\n\n### Kafka Schema\n\nFlow exporter requires a Kafka topic that has events which contain the following JSON attributes:\n\n```json\n{\n  \"label\": \"bdr1.fqdn.com\",\n  \"as_src\": 15169,\n  \"as_dst\": 6939,\n  \"bytes\": 52,\n}\n```\n\n- `label`: The hostname of the device that the flow came from\n- `as_src`: The ASN that originated the flow\n- `as_dst`: The ASN that the flow was destined for\n- `bytes`: The number of bytes contained in the flow\n\n## [pmacct](https://github.com/pmacct/pmacct) Integration\n\nFlow Exporter works well with pmacct, a series of tools for monitoring flows in Linux. The following pmacctd configuration can be used to collect flows on Linux, enrich them with BGP ASN data, and publish them to Kafka:\n\n#### `/etc/pmacct/pmacctd.conf`\n\n```\n!\n! pmacctd configuration example\n!\n! Did you know CONFIG-KEYS contains the detailed list of all configuration keys\n! supported by 'nfacctd' and 'pmacctd' ?\n!\n! debug: true\ndaemonize: false\npcap_interfaces_map: /etc/pmacct/interfaces.map\npmacctd_as: longest\npmacctd_net: longest\nsampling_rate: 1\n!\nbgp_daemon: true\nbgp_daemon_ip: 127.0.0.2\nbgp_daemon_port: 180\nbgp_daemon_max_peers: 10\nbgp_agent_map: /etc/pmacct/peering_agent.map\nnetworks_file: /etc/pmacct/networks.lst\nnetworks_file_no_lpm: true\n!\naggregate: src_host, dst_host, src_port, dst_port, src_as, dst_as, label\n!\nplugins: kafka\nkafka_output: json\nkafka_broker_host: kafka.fqdn.com\nkafka_topic: pmacct.acct\nkafka_refresh_time: 5\nkafka_history: 5m\nkafka_history_roundoff: m\n```\n\nAnd the associated configurations referenced in that file:\n\n#### `/etc/pmacct/interfaces.map`\n\n```\nifindex=100 ifname=\u003cINTERFACE\u003e\n```\n\n#### `/etc/pmacct/peering_agent.map`\n\n```\nbgp_ip=\u003cBGP_ROUTER_ID\u003e     ip=0.0.0.0/0\n```\n\nMore information on configuring pmacct can be found [here](https://github.com/pmacct/pmacct/blob/master/CONFIG-KEYS).\n\n## Docker\n\nA Dockerfile is provided for convenience. It will build the source and then run the exporter. You can use the Docker command line like so:\n\n```\ndocker run -p 9590:9590 bswinnerton/flow-exporter:latest --brokers=kafka.fqdn.com:9092 --topic=pmacct.acct --asn=15169\n```\n\nOr if you prefer Docker Compose:\n\n```yml\nflow-exporter:\n  image: bswinnerton/flow-exporter:latest\n  command: --brokers=kafka.fqdn.com:9092 --topic=pmacct.acct --asn=15169\n  expose:\n    - 9590\n```\n\nIdeally in the same `docker-compose.yml` file as your Prometheus server to make communication easy.\n\n## Building\n\nThe application can be compiled by running:\n\n```\ngit clone https://github.com/neptune-networks/flow-exporter\ncd flow-exporter/cmd/flow-exporter\ngo build main.go\n```\n\n## Releasing\n\nTo release a new version, the following commands must be run:\n\n```\ngit tag -a vX.Y.Z -m \"vX.Y.Z\"\ngit push origin vX.Y.Z\ngoreleaser --rm-dist\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneptune-networks%2Fflow-exporter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fneptune-networks%2Fflow-exporter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneptune-networks%2Fflow-exporter/lists"}