{"id":13581998,"url":"https://github.com/netsampler/goflow2","last_synced_at":"2025-12-27T09:47:06.493Z","repository":{"id":36955499,"uuid":"355031433","full_name":"netsampler/goflow2","owner":"netsampler","description":"High performance sFlow/IPFIX/NetFlow Collector","archived":false,"fork":false,"pushed_at":"2025-05-03T17:32:05.000Z","size":498,"stargazers_count":593,"open_issues_count":11,"forks_count":127,"subscribers_count":15,"default_branch":"main","last_synced_at":"2025-05-03T18:29:22.984Z","etag":null,"topics":["collector","go","ipfix","kafka","netflow","network","protobuf","sflow"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/netsampler.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-04-06T02:22:57.000Z","updated_at":"2025-05-03T17:32:08.000Z","dependencies_parsed_at":"2023-10-02T02:34:37.322Z","dependency_job_id":"bc3df4fe-1e6d-4ec3-a95c-9bcd995e7113","html_url":"https://github.com/netsampler/goflow2","commit_stats":{"total_commits":174,"total_committers":23,"mean_commits":7.565217391304348,"dds":"0.45977011494252873","last_synced_commit":"34ab2b54598e3a771fb621929a7b1ad9406908a3"},"previous_names":[],"tags_count":26,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/netsampler%2Fgoflow2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/netsampler%2Fgoflow2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/netsampler%2Fgoflow2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/netsampler%2Fgoflow2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/netsampler","download_url":"https://codeload.github.com/netsampler/goflow2/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254000851,"owners_count":21997441,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["collector","go","ipfix","kafka","netflow","network","protobuf","sflow"],"created_at":"2024-08-01T15:02:22.496Z","updated_at":"2025-12-27T09:47:06.486Z","avatar_url":"https://github.com/netsampler.png","language":"Go","readme":"# GoFlow2\n\n[![Build Status](https://github.com/netsampler/goflow2/workflows/Build/badge.svg)](https://github.com/netsampler/goflow2/actions?query=workflow%3ABuild)\n[![Go Reference](https://pkg.go.dev/badge/github.com/netsampler/goflow2.svg)](https://pkg.go.dev/github.com/netsampler/goflow2)\n\nThis application is a NetFlow/IPFIX/sFlow collector in Go.\n\nIt gathers network information (IP, interfaces, routers) from different flow protocols,\nserializes it in a common format.\n\nYou will want to use GoFlow if:\n* You receive a decent amount of network samples and need horizontal scalability\n* Have protocol diversity and need a consistent format\n* Require raw samples and build aggregation and custom enrichment\n\nThis software is the entry point of a pipeline. The storage, transport, enrichment, graphing, alerting are\nnot provided.\n\n![GoFlow2 System diagram](/graphics/diagram.png)\n\n## Origins\n\nThis work is a fork of a previous [open-source GoFlow code](https://github.com/cloudflare/goflow) built and used at Cloudflare.\nIt lives in its own GitHub organization to be maintained more easily.\n\nAmong the differences with the original code:\nThe serializer and transport options have been revamped to make this program more user-friendly\nand target new use-cases like logging providers.\nMinimal changes in the decoding libraries.\n\n## Modularity\n\nIn order to enable load-balancing and optimizations, the GoFlow2 library has a `decoder` which converts\nthe payload of a flow packet into a structure.\n\nThe `producer` converts the samples into another format.\nOut of the box, this repository provides a protobuf producer (`pb/flow.pb`)\nand a raw producer.\nIn the case of the protobuf producer, the records in a single flow packet\nare extracted and made in their own protobuf. Custom mapping allows\nto add new fields without rebuilding the proto.\n\nThe `format` directory offers various utilities to format a message. It calls specific\nfunctions to marshal as JSON or text for instance.\n\nThe `transport` provides different way of processing the message. Either sending it via Kafka or \nsend it to a file (or stdout).\n\nGoFlow2 is a wrapper of all the functions and chains them.\n\nYou can build your own collector using this base and replace parts:\n* Use different transport (e.g: RabbitMQ instead of Kafka)\n* Convert to another format (e.g: Cap'n Proto, Avro, instead of protobuf)\n* Decode different samples (e.g: not only IP networks, add MPLS)\n* Different metrics system (e.g: [OpenTelemetry](https://opentelemetry.io/))\n\n### Protocol difference\n\nThe sampling protocols have distinct features:\n\n**sFlow** is a stateless protocol which sends the full header of a packet with router information\n(interfaces, destination AS) while **NetFlow/IPFIX** rely on templates that contain fields (e.g: source IPv6).\n\nThe sampling rate in NetFlow/IPFIX is provided by **Option Data Sets**. This is why it can take a few minutes\nfor the packets to be decoded until all the templates are received (**Option Template** and **Data Template**).\n\nBoth of these protocols bundle multiple samples (**Data Set** in NetFlow/IPFIX and **Flow Sample** in sFlow)\nin one packet.\n\nThe advantages of using an abstract network flow format, such as protobuf, is it enables summing over the\nprotocols (e.g: per ASN or per port, rather than per (ASN, router) and (port, router)).\n\nTo read more about the protocols and how they are mapped inside, check out [page](/docs/protocols.md)\n\n### Features of GoFlow2\n\nCollection:\n* NetFlow v5\n* IPFIX/NetFlow v9 (sampling rate provided by the Option Data Set)\n* sFlow v5\n\n(adding NetFlow v1,7,8 is being evaluated)\n\nProduction:\n* Convert to protobuf or json\n* Prints to the console/file\n* Sends to Kafka and partition\n\nMonitoring via Prometheus metrics\n\n## Get started\n\nTo read about agents that samples network traffic, check this [page](/docs/agents.md).\n\nTo set up the collector, download the latest release corresponding to your OS\nand run the following command (the binaries have a suffix with the version):\n\n```bash\n$ ./goflow2\n```\n\nBy default, this command will launch an sFlow collector on port `:6343` and\na NetFlowV9/IPFIX collector on port `:2055`.\n\nBy default, the samples received will be printed in JSON format on the stdout.\n\n```json\n{\n    \"type\": \"SFLOW_5\",\n    \"time_received_ns\": 1681583295157626000,\n    \"sequence_num\": 2999,\n    \"sampling_rate\": 100,\n    \"sampler_address\": \"192.168.0.1\",\n    \"time_flow_start_ns\": 1681583295157626000,\n    \"time_flow_end_ns\": 1681583295157626000,\n    \"bytes\": 1500,\n    \"packets\": 1,\n    \"src_addr\": \"fd01::1\",\n    \"dst_addr\": \"fd01::2\",\n    \"etype\": \"IPv6\",\n    \"proto\": \"TCP\",\n    \"src_port\": 443,\n    \"dst_port\": 50001\n}\n```\n\nIf you are using a log integration (e.g: Loki with Promtail, Splunk, Fluentd, Google Cloud Logs, etc.),\njust send the output into a file.\n\n```bash\n$ ./goflow2 -transport.file /var/logs/goflow2.log\n```\n\nTo enable Kafka and send protobuf, use the following arguments:\n\n```bash\n$ ./goflow2 -transport=kafka \\\n  -transport.kafka.brokers=localhost:9092 \\\n  -transport.kafka.topic=flows \\\n  -format=bin\n```\n\nBy default, the distribution will be randomized.\nIn order to partition the field, you need to configure the `key`\nin the formatter.\n\nBy default, compression is disabled when sending data to Kafka.\nTo change the kafka compression type of the producer side configure the following option:\n\n```\n-transport.kafka.compression.type=gzip\n```\nThe list of codecs is available in the [Sarama documentation](https://pkg.go.dev/github.com/Shopify/sarama#CompressionCodec).\n\n\nBy default, the collector will listen for IPFIX/NetFlow V9/NetFlow V5 on port 2055\nand sFlow on port 6343.\nTo change the sockets binding, you can set the `-listen` argument and a URI\nfor each protocol (`netflow`, `sflow` or `flow` for both as scheme) separated by a comma.\nFor instance, to create 4 parallel sockets of sFlow and one of NetFlow, you can use:\n\n```bash\n$ ./goflow2 -listen 'sflow://:6343?count=4,netflow://:2055'\n```\n\nMore information about workers and resource usage is available on the [Performance page](/docs/performance.md).\n\n### Docker\n\nYou can also run directly with a container:\n```\n$ sudo docker run -p 6343:6343/udp -p 2055:2055/udp -ti netsampler/goflow2:latest\n```\n\n### Mapping extra fields\n\nIn the case of exotic template fields or extra payload not supported by GoFlow2\nof out the box, it is possible to pass a mapping file using `-mapping mapping.yaml`.\nA [sample file](cmd/goflow2/mapping.yaml) is available in the `cmd/goflow2` directory.\n\nFor instance, certain devices producing IPFIX use `ingressPhysicalInterface` (id: 252)\nand do not use `ingressInterface` (id: 10). Using the following you can have the interface mapped\nin the InIf protobuf field without changing the code.\n\n```yaml\nipfix:\n  mapping:\n    - field: 252\n      destination: in_if\n    - field: 253\n      destination: out_if\n```\n\n### Output format considerations\n\nThe JSON format is advised only when consuming a small amount of data directly.\nFor bigger workloads, the protobuf output format provides a binary representation\nand is preferred.\nIt can also be extended with enrichment as long as the user keep the same IDs.\n\nIf you want to develop applications, build `pb/flow.proto` into the language you want:\nWhen adding custom fields, picking a field ID ≥ 1000 is suggested.\n\nCheck the docs for more information about [compiling protobuf](/docs/protobuf.md). \n\n## Flow Pipeline\n\nA basic enrichment tool is available in the `cmd/enricher` directory.\nYou need to load the Maxmind GeoIP ASN and Country databases using `-db.asn` and `-db.country`.\n\nRunning a flow enrichment system is as simple as a pipe.\nOnce you plug the stdin of the enricher to the stdout of GoFlow in protobuf,\nthe source and destination IP addresses will automatically be mapped \nwith a database for Autonomous System Number and Country.\nSimilar output options as GoFlow are provided.\n\n```bash\n$ ./goflow2 -transport.file.sep= -format=bin | \\\n  ./enricher -db.asn path-to/GeoLite2-ASN.mmdb -db.country path-to/GeoLite2-Country.mmdb\n```\n\nFor a more scalable production setting, Kafka and protobuf are recommended.\nStream operations (aggregation and filtering) can be done with stream-processor tools.\nFor instance Flink, or the more recent Kafka Streams and kSQLdb.\nDirect storage can be done with data-warehouses like Clickhouse.\n\nEach protobuf message is prefixed by its varint length.\n\nThis repository contains [examples of pipelines](./compose) with docker-compose.\nThe available pipelines are:\n* [Kafka+Clickhouse+Grafana](./compose/kcg)\n* [Logstash+Elastic+Kibana](./compose/elk)\n\n## Security notes and assumptions\n\nBy default, the buffer for UDP is 9000 bytes.\nProtections were added to avoid DOS on sFlow since the various length fields are 32 bits.\nThere are assumptions on how many records and list items a sample can have (eg: AS-Path).\n\n## User stories\n\nAre you using GoFlow2 in production at scale? Add yourself here!\n\n### Contributions\n\nThis project welcomes pull-requests, whether it's documentation,\ninstrumentation (e.g: docker-compose, metrics), internals (protocol libraries),\nintegration (new CLI feature) or else!\nJust make sure to check for the use-cases via an issue.\n\nThis software would not exist without the testing and commits from\nits users and [contributors](docs/contributors.md).\n\n## License\n\nLicensed under the BSD-3 License.\n","funding_links":[],"categories":["Go"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnetsampler%2Fgoflow2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnetsampler%2Fgoflow2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnetsampler%2Fgoflow2/lists"}