{"id":31581025,"url":"https://github.com/wide-vsix/linux-flow-exporter","last_synced_at":"2025-10-05T21:52:15.334Z","repository":{"id":40472776,"uuid":"507558574","full_name":"wide-vsix/linux-flow-exporter","owner":"wide-vsix","description":"eBPF based IPFIX exporter. This software is an IPFIX flow-exporter for routing with Linux kernel. It records flow stats forwarded by the kernel using tc-ebpf, without AF_PACKET or conntrack. This flow-exporter supports multipath environment with less performance issues.","archived":false,"fork":false,"pushed_at":"2023-06-24T03:25:42.000Z","size":317,"stargazers_count":33,"open_issues_count":1,"forks_count":6,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-06-19T16:48:58.779Z","etag":null,"topics":["ebpf","ipfix","linux"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wide-vsix.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-26T11:43:33.000Z","updated_at":"2024-05-20T20:47:14.000Z","dependencies_parsed_at":"2024-06-19T16:25:00.303Z","dependency_job_id":"2c396248-add1-467f-b177-da14e1d3067c","html_url":"https://github.com/wide-vsix/linux-flow-exporter","commit_stats":{"total_commits":40,"total_committers":3,"mean_commits":"13.333333333333334","dds":"0.050000000000000044","last_synced_commit":"cb0706803b2be85a9b35d39fcd29c7978dac46a2"},"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/wide-vsix/linux-flow-exporter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wide-vsix%2Flinux-flow-exporter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wide-vsix%2Flinux-flow-exporter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wide-vsix%2Flinux-flow-exporter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wide-vsix%2Flinux-flow-exporter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wide-vsix","download_url":"https://codeload.github.com/wide-vsix/linux-flow-exporter/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wide-vsix%2Flinux-flow-exporter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278526242,"owners_count":26001325,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-05T02:00:06.059Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ebpf","ipfix","linux"],"created_at":"2025-10-05T21:52:14.314Z","updated_at":"2025-10-05T21:52:15.328Z","avatar_url":"https://github.com/wide-vsix.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# linux-flow-exporter\n\neBPF based IPFIX exporter. This software is an IPFIX flow-exporter for routing\nwith Linux kernel. It records flow stats forwarded by the kernel using tc-ebpf,\nwithout AF_PACKET or conntrack. Some flow-exporter using AF_PACKET has\nperformance issues due to frequent user/kernel communication, and Some one using\nconntrack does not work properly in a multipath environment. This software is an\nflow-exporter that does not have such issues and supports multipath environment\nwith less performance issues.\n\n## System Components\n\n- ebpflow: in-kernel flow-stats collector with ebpf\n- flowctl: user-space cli utility includes:\n  - daemonized agent for IPFIX flow-exporter\n  - dump the in-kernel flow-stats from the user-space\n  - IPFIX dummy data transmitter for test\n- dependencies (tested)\n  - linux kernel 5.x+\n  - iproute2 5.18+\n\nRequirements: you can verifiy compatibity with `dependency-check` subcmd.\n- `clang --version`: 10.0.0\n- `uname -r`: 5.15.0-1008-gcp\n- `ip -V`: ip utility, iproute2-5.18.0, libbpf 0.8.0\n  - iproute2 is needed to attach ebpf program into the kernel.\n\n```\n$ sudo flowctl dependency-check\nclang version (expect v10.0.0): v12.0.1 (VALID)\nkernel version (expect v5.4.0): v5.15.0 (VALID)\niproute2 binary version (expect v5.4.0): v5.18.0 (VALID)\niproute2 libbpf version (expect v0.8.0): v0.8.0 (VALID)\n```\n\n![](./img/linux_datapath.drawio.svg)\n\n## Example Usage\n\n```yaml\ncollectors:\n- address: 10.146.0.6:2100\ntemplates:\n- id: 1024\n  template:\n  - FlowEndMilliseconds\n  - FlowStartMilliseconds\n  - OctetDeltaCount\n  - PacketDeltaCount\n  - IpVersion\n  - FlowDirection\n  - SourceIPv4Address\n  - DestinationIPv4Address\n```\n\n```\ngit clone \u003cthis-repo\u003e\ncd \u003cthis-repo\u003e\nsudo ./misc/create_netns.sh\nsudo flowctl meter attach --netns ns0 -n eth1\nsudo flowctl meter attach --netns ns0 -n eth2\nsudo flowctl meter attach --netns ns0 -n eth3\n```\n\n### flowctl usage\n\n```shell\ndocker run --rm --name tmp -it --privileged --net=host -v /usr/include/asm:/usr/include/asm -v /var/run/netns:/var/run/netns ghcr.io/wide-vsix/linux-flow-exporter:branch-master bash\nflowctl meter status\n```\n\n```shell\n## How to check current flow cache\n$ sudo flowctl dump\nIFINDEX  PROTO  SRC               DST               PKTS  BYTES\n98       6      172.17.0.7:49375  172.67.134.3:80   1707  186818\n98       6      172.17.0.7:41585  104.21.25.104:80  1710  187560\n98       6      172.17.0.7:37869  104.21.25.104:80  9     486\n\n$ sudo flowctl flush -i 98 -p 6 -s 172.17.0.7 -S 37869 -d 104.21.25.104 -D 80 # one cache\n$ sudo flowctl flush --all # all caches\n```\n\n## Limitation\n\nThis software works ONLY for tcp.\n\n## Utilities\n\n```\ndocker run --rm -td --name tmp1 nicolaka/netshoot bash\n```\n\n## Background Why we need\n\n- conntrack doesn't support async traffic\n- libpcap based approach consume extreamly big computing cost\n\nREFS\n- [Let's read RFC regarding IPFIX (ja) by Asama-san](https://enog.jp/wordpress/wp-content/uploads/2011/12/ipfix.pdf)\n- [SKB Definition](https://elixir.bootlin.com/linux/latest/source/include/linux/skbuff.h)\n- [Connection Tracking (conntrack): Design and Implementation Inside Linux Kernel](https://arthurchiao.art/blog/conntrack-design-and-implementation/)\n- [Packet mark in a Cloud Native world, LPC](https://lpc.events/event/7/contributions/683/attachments/554/979/lpc20-pkt-mark-slides.pdf)\n- [VMware NSX IPFIX for Distributed Firewall](https://docs.vmware.com/en/VMware-NSX-Data-Center-for-vSphere/6.4/com.vmware.nsx.admin.doc/GUID-2C625B52-17F0-4604-B5C9-6DF1FA9A70F8.html)\n- [VMware NSX IPFIX for Logical Switch](https://docs.vmware.com/en/VMware-NSX-Data-Center-for-vSphere/6.4/com.vmware.nsx.admin.doc/GUID-6054CF07-3019-4539-A6CC-1F613E275E27.html)\n- [Comparison and Practice of packet processing implementations and acceleration methods (ja), Ebiken-san, Higebu-san, JANOG45](https://www.janog.gr.jp/meeting/janog45/program/pktfwd/)\n- [IN-kernel metadata propagation technique from XDP buffer to SKB](https://github.com/torvalds/linux/blob/master/samples/bpf/xdp2skb_meta_kern.c)\n- [Private Discussion for metadata practice in eBPF](https://netmoles.slack.com/archives/C01DYBEETN0/p1667363998874089)\n- [One of the Reference design for traffic control mech using eBPF(both xdp and tc), ENOG63 by Higebu-san](https://www.docswell.com/s/higebu/ZQWRMZ-xdp-based-mobile-network-data-plane#p21)\n- [An open, unofficial registry of linux packet mark bits (aka fwmark, connmark, netfilter, iptables, nftables)](https://github.com/fwmark/registry)\n- [BPF Features by Linux Kernel Version](https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md)\n\n## Specification\n\n### Supported Text Log Keys\n\nsupported\n```\nsrc, string\ndst, string\nproto, string\nstarttime\nendtime\npkts\nbytes\n```\n\nconsideration to support\n```\nmatched acl rule number\n```\n\n### Supported IPFIX IETF IE\n\nreference: [IANA registration](https://www.iana.org/assignments/ipfix/ipfix.xhtml)\n\n```\n  {\"FlowEndMilliseconds\", netflow.IPFIX_FIELD_flowEndMilliseconds, 8},\n  {\"FlowStartMilliseconds\", netflow.IPFIX_FIELD_flowStartMilliseconds, 8},\n  {\"FlowEndNanoseconds\", netflow.IPFIX_FIELD_flowEndNanoseconds, 8},\n  {\"FlowStartNanoseconds\", netflow.IPFIX_FIELD_flowStartNanoseconds, 8},\n  {\"OctetDeltaCount\", netflow.IPFIX_FIELD_octetDeltaCount, 8},\n  {\"PacketDeltaCount\", netflow.IPFIX_FIELD_packetDeltaCount, 8},\n  {\"IpVersion\", netflow.IPFIX_FIELD_ipVersion, 1},\n  {\"IngressInterface\", netflow.IPFIX_FIELD_ingressInterface, 4},\n  {\"EgressInterface\", netflow.IPFIX_FIELD_egressInterface, 4},\n  {\"SourceIPv4Address\", netflow.IPFIX_FIELD_sourceIPv4Address, 4},\n  {\"DestinationIPv4Address\", netflow.IPFIX_FIELD_destinationIPv4Address, 4},\n  {\"ProtocolIdentifier\", netflow.IPFIX_FIELD_protocolIdentifier, 1},\n  {\"SourceTransportPort\", netflow.IPFIX_FIELD_sourceTransportPort, 2},\n  {\"DestinationTransportPort\", netflow.IPFIX_FIELD_destinationTransportPort, 2},\n```\n\nunder the development\n```\n  {\"forwardingStatus\", 89, 1}\n```\n\nfollow will be supported, in mid term\n```\n  {\"flowDirection\", netflow.IPFIX_FIELD_flowDirection, 1},\n  {\"tcpControlBits\", netflow.IPFIX_FIELD_tcpControlBits, 1},\n  {\"icmpTypeCodeIPv4\", netflow.IPFIX_FIELD_icmpTypeCodeIPv4, 2},\n```\n\nfollow will be supported, in long term\n```\n  {\"ipClassOfService\", netflow.IPFIX_FIELD_ipClassOfService, 1},\n  {\"sourceIPv4PrefixLength\", netflow.IPFIX_FIELD_sourceIPv4PrefixLength, 1},\n  {\"destinationIPv4PrefixLength\", netflow.IPFIX_FIELD_destinationIPv4PrefixLength, 1},\n  {\"ipNextHopIPv4Address\", netflow.IPFIX_FIELD_ipNextHopIPv4Address, 4},\n  {\"bgpSourceAsNumber\", netflow.IPFIX_FIELD_bgpSourceAsNumber, 4},\n  {\"bgpDestinationAsNumber\", netflow.IPFIX_FIELD_bgpDestinationAsNumber, 4},\n  {\"bgpNextHopIPv4Address\", netflow.IPFIX_FIELD_bgpNextHopIPv4Address, 4},\n  {\"minimumTTL\", netflow.IPFIX_FIELD_minimumTTL, 1},\n  {\"maximumTTL\", netflow.IPFIX_FIELD_maximumTTL, 1},\n  {\"fragmentIdentification\", netflow.IPFIX_FIELD_fragmentIdentification, 4},\n  {\"vlanId\", netflow.IPFIX_FIELD_vlanId, 2},\n  {\"flowEndReason\", netflow.IPFIX_FIELD_flowEndReason, 1},\n  {\"dot1qVlanId\", netflow.IPFIX_FIELD_dot1qVlanId, 2},\n  {\"dot1qCustomerVlanId\", netflow.IPFIX_FIELD_dot1qCustomerVlanId, 2},\n```\n\n\n### Supported IPFIX Enterprise IE\n\nEnterprise No: 28972\n([Keio University, iana registry](https://www.iana.org/assignments/enterprise-numbers/enterprise-numbers)).\nIt may be updated by LINE Corporation\n\n```\n```\n\n## Pre Build Packages\n\n![](https://github.com/wide-vsix/linux-flow-exporter/actions/workflows/binary_branch.yaml/badge.svg)\u003cbr/\u003e\n![](https://github.com/wide-vsix/linux-flow-exporter/actions/workflows/binary_release.yaml/badge.svg)\u003cbr/\u003e\n![](https://github.com/wide-vsix/linux-flow-exporter/actions/workflows/image_branch.yaml/badge.svg)\u003cbr/\u003e\n![](https://github.com/wide-vsix/linux-flow-exporter/actions/workflows/image_release.yaml/badge.svg)\u003cbr/\u003e\n\nYou can use\n[automated build container images](https://github.com/wide-vsix/linux-flow-exporter/pkgs/container/linux-flow-exporter/versions?filters%5Bversion_type%5D=tagged) or\n[automated build elf-binary](https://github.com/wide-vsix/linux-flow-exporter/releases).\nBoth are automatically created by updating main-branch and creating release.\nPlease note that flowctl depends on clang, iproute2 and linux kernel.\nYou can use `flowctl dependency-check` to check for dependency problems.\n\nfor container (latest version)\n```\ndocker run -it --rm --privileged --net=host \\\n  -v /usr/include/asm:/usr/include/asm \\\n  -v /var/run/netns:/var/run/netns \\\n  ghcr.io/wide-vsix/linux-flow-exporter:branch-main bash\nflowctl version\n```\n\nfor container (specified version)\n```\ndocker run -it --rm --privileged --net=host \\\n  -v /usr/include/asm:/usr/include/asm \\\n  -v /var/run/netns:/var/run/netns \\\n  ghcr.io/wide-vsix/linux-flow-exporter:v0.0.5 bash\nflowctl version\n```\n\nfor binary (latest version)\n```\ncurl -Lo /usr/local/bin/flowctl https://github.com/wide-vsix/linux-flow-exporter/releases/download/branch-main/flowctl.linux-amd64\nchmod +x /usr/local/bin/flowctl\nflowctl version\n```\n\nfor binary (specified version)\n```\ncurl -Lo /usr/local/bin/flowctl https://github.com/wide-vsix/linux-flow-exporter/releases/download/v0.0.5/flowctl.linux-amd64\nchmod +x /usr/local/bin/flowctl\nflowctl version\n```\n\n## Licence\n\nThe user space components are licensed under the\n[Apache License, Version 2.0](https://github.com/wide-vsix/linux-flow-exporter/blob/main/LICENSE).\nThe BPF code templates are dual-licensed under the\n[General Public License](https://github.com/wide-vsix/linux-flow-exporter/blob/main/bpf/LICENSE.GPL-2.0),\nVersion 2.0 (only) and the\n[2-Clause BSD License](https://github.com/wide-vsix/linux-flow-exporter/blob/main/bpf/LICENSE.BSD-2-Clause)\n(you can use the terms of either license, at your option).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwide-vsix%2Flinux-flow-exporter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwide-vsix%2Flinux-flow-exporter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwide-vsix%2Flinux-flow-exporter/lists"}