{"id":15057071,"url":"https://github.com/BelWue/flowpipeline","last_synced_at":"2025-10-31T18:30:39.561Z","repository":{"id":38822798,"uuid":"392673655","full_name":"BelWue/flowpipeline","owner":"BelWue","description":"Process network flows using fully configurable pipelines.","archived":false,"fork":false,"pushed_at":"2025-02-03T09:17:25.000Z","size":743,"stargazers_count":21,"open_issues_count":6,"forks_count":7,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-02-15T06:14:13.365Z","etag":null,"topics":["ebpf","go","goflow","golang","influxdb","kafka","monitoring","netflow","network-analysis","networking","prometheus","tcpdump-like"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BelWue.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-08-04T12:06:07.000Z","updated_at":"2025-02-03T09:09:07.000Z","dependencies_parsed_at":"2024-12-16T16:32:20.661Z","dependency_job_id":"55feb9fc-1647-4e1e-b386-74515363f459","html_url":"https://github.com/BelWue/flowpipeline","commit_stats":{"total_commits":198,"total_committers":8,"mean_commits":24.75,"dds":0.3535353535353535,"last_synced_commit":"02e9b61221aa64e5feadce9a00cd8b6a2994d2cb"},"previous_names":["belwue/flowpipeline","bwnetflow/flowpipeline"],"tags_count":26,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BelWue%2Fflowpipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BelWue%2Fflowpipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BelWue%2Fflowpipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BelWue%2Fflowpipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BelWue","download_url":"https://codeload.github.com/BelWue/flowpipeline/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239223748,"owners_count":19602849,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ebpf","go","goflow","golang","influxdb","kafka","monitoring","netflow","network-analysis","networking","prometheus","tcpdump-like"],"created_at":"2024-09-24T22:01:42.020Z","updated_at":"2025-10-31T18:30:39.506Z","avatar_url":"https://github.com/BelWue.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Flow PipelineSrcAS\n\n[godoc](https://pkg.go.dev/github.com/BelWue/flowpipeline)\n\n## About The Project\n\n[bwNET](https://bwnet.belwue.de/) is a research project of the German federal\nstate of Baden-Württemberg which aims to provide innovative services within the\nstate's research and education network [BelWü](https://www.belwue.de). This\nGitHub Org contains the code pertaining to the monitoring aspect of this\nproject.\n\nThis repo contains our flow processing toolkit which enables us and our users\nto define pipelines for [goflow2](https://github.com/netsampler/goflow2)-compatible\nflow messages. The flowpipeline project integrates most other parts of our flow\nprocessing stack into a single piece of software which can be configured to\nserve any function:\n\n* accepting raw Netflow (using [goflow2](https://github.com/netsampler/goflow2))\n* enriching the resulting flow messages ([examples/enricher](https://github.com/BelWue/flowpipeline/tree/master/examples/enricher))\n* writing to and reading from Kafka ([examples/localkafka](https://github.com/BelWue/flowpipeline/tree/master/examples/localkafka))\n* dumping flows to cli (e.g. [flowdump](https://github.com/BelWue/flowpipeline/tree/master/examples/flowdump))\n* providing metrics and insights ([examples/prometheus](https://github.com/BelWue/flowpipeline/tree/master/examples/prometheus))\n* and many more...\n\n## Getting Started\n\nTo get going, choose one of the following deployment methods.\n\n### Compile from Source\nClone this repo and use `go build .` to build the binary yourself.\n\nBy default, the binary will look for a `config.yml` in its local directory, so\nyou'll either want to create one or call it from any example directory (and\nmaybe follow the instructions there).\n\n### Binary Releases\nDownload our [latest release](https://github.com/BelWue/flowpipeline/releases)\nand run it, same as if you compiled it yourself.\n\nThe default, dynamically linked version requires a reasonably recent system\n(glibc 2.32+, linux 5.11+ for `bpf`, `mongodb` ...) and comes with all features.\nAs a fallback option, the static binaries will work in older environments\n(CentOS 7, Debian 10, ...), but come without the segments that require\nCGO/dynamically linked code (`bpf`, `sqlite`, `mongodb` and plugin support, check\n[CONFIGURATION.md](https://github.com/BelWue/flowpipeline/blob/master/CONFIGURATION.md)).\n\n### Container Releases\nA ready to use container is provided as `bwnetflow/flowpipeline`, you can check\nit out on [GitHub container registry](https://github.com/BelWue/flowpipeline/pkgs/container/flowpipeline).\n\nConfigurations referencing other files (geolocation databases for instance)\nwill work in a container without extra edits. This is because the volume\nmountpoint `/config` is prepended in all segments which accept configuration to\nopen files, if the binary was built with the `container` build flag.\n\n```sh\npodman run -v ./examples/xy:/config flowpipeline\n# or\ndocker run -v ./examples/xy:/config flowpipeline\n```\n\n## Configuration\n\nRefer to [CONFIGURATION.md](https://github.com/BelWue/flowpipeline/blob/master/CONFIGURATION.md)\nfor the full guide. Other than that, looking at the examples should give you a\ngood idea what the config looks like in detail and what the possible\napplications are. For sake of completeness, here's another minimal example\nwhich starts listening for Netflow v9 on port 2055, applies the filter given as\nfirst argument, and then prints it out in a `tcpdump`-style format.\n\n```yaml\n- segment: goflow\n- segment: flowfilter\n  config:\n    filter: $0\n- segment: printflowdump\n```\n\nYou'd call it with `./flowpipeline \"proto tcp and (port 80 or port 443)\"`., for\ninstance.\n\n### Custom Segments\nIf you find that the existing segments lack some functionality or you require\nsome very specific behaviour, it is possible to include segments as a plugin.\nThis is done using the `-p yourplugin.so` commandline option and your own\ncustom module. See\n[examples/plugin](https://github.com/BelWue/flowpipeline/tree/master/examples/plugin)\nfor a basic example and instructions on how to compile your plugin.\n\nNote that this requires CGO and thus will not work using the static binary\nreleases or in a container.\n\n## Contributing\n\nContributions in any form (code, issues, feature requests) are very much welcome.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBelWue%2Fflowpipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FBelWue%2Fflowpipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBelWue%2Fflowpipeline/lists"}