{"id":49341731,"url":"https://github.com/redborder/f2k","last_synced_at":"2026-04-27T04:04:51.863Z","repository":{"id":143877245,"uuid":"71358028","full_name":"redBorder/f2k","owner":"redBorder","description":"netflow 2 kafka translator","archived":false,"fork":false,"pushed_at":"2023-11-08T12:16:16.000Z","size":812,"stargazers_count":20,"open_issues_count":9,"forks_count":4,"subscribers_count":7,"default_branch":"master","last_synced_at":"2023-11-08T13:30:15.292Z","etag":null,"topics":["redborder","redborder-ng","rpm","service"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/redBorder.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2016-10-19T13:05:36.000Z","updated_at":"2023-11-08T13:30:21.334Z","dependencies_parsed_at":"2023-11-08T13:30:21.287Z","dependency_job_id":"30848022-b51e-40fe-8bcc-1e3734d174d3","html_url":"https://github.com/redBorder/f2k","commit_stats":null,"previous_names":[],"tags_count":6,"template":null,"template_full_name":null,"purl":"pkg:github/redBorder/f2k","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redBorder%2Ff2k","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redBorder%2Ff2k/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redBorder%2Ff2k/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redBorder%2Ff2k/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/redBorder","download_url":"https://codeload.github.com/redBorder/f2k/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/redBorder%2Ff2k/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32321945,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-26T23:26:28.701Z","status":"online","status_checked_at":"2026-04-27T02:00:06.769Z","response_time":128,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["redborder","redborder-ng","rpm","service"],"created_at":"2026-04-27T04:04:51.701Z","updated_at":"2026-04-27T04:04:51.853Z","avatar_url":"https://github.com/redBorder.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/redBorder/f2k.svg?branch=master)](https://travis-ci.org/redBorder/f2k)\n[![Coverage Status](https://coveralls.io/repos/github/redBorder/f2k/badge.svg?branch=master)](https://coveralls.io/github/redBorder/f2k?branch=master)\n\n# Flow 2 Kafka (f2k)\n\n* [Setup](#setup)\n* [Usage](#usage)\n   * [Basic usage](#basic-usage)\n   * [Sensors config](#sensors-config)\n* [Others configuration parameters](#others-configuration-parameters)\n   * [Multi-thread](#multi-thread)\n   * [Long flow separation](#long-flow-separation)\n   * [Geo information](#geo-information)\n   * [Names resolution](#names-resolution)\n   * [Mac vendor information (mac_vendor)](#mac-vendor-information-mac_vendor)\n   * [Applications/engine ID (applications, \u003ccode\u003eengines\u003c/code\u003e)](#applicationsengine-id-applications-engines)\n   * [Hosts, domains, vlan (hosts, \u003ccode\u003ehttp_domains\u003c/code\u003e, \u003ccode\u003evlans\u003c/code\u003e)](#hosts-domains-vlan-hosts-http_domains-vlans)\n   * [Netflow probe nets](#netflow-probe-nets)\n   * [DNS](#dns)\n   * [Template cache](#template-cache)\n      * [Using folder](#using-folder)\n      * [Using \u003ca href=\"https://zookeeper.apache.org/\"\u003eApache zookeeper\u003c/a\u003e](#using-apache-zookeeper)\n   * [\u003ca href=\"https://github.com/edenhill/librdkafka\"\u003elibrdkafka\u003c/a\u003e options](#librdkafka-options)\n\nNetflow to\n[Json](http://www.json.org/)/[Kafka](https://kafka.apache.org/) collector.\n\n## Setup\nTo use it, you only need to do a typical `./configure \u0026\u0026 make \u0026\u0026 make install`\n\n## Usage\n\n### Basic usage\n\nThe most important configuration parameters are:\n\n- Output parameters:\n    - `--kafka=127.0.0.1@rb_flow`, broker@topic to send netflow\n\n- Input parameters: Can be either UDP port or Kafka topic\n    - `--collector-port=2055`, Collector port to listen netflow\n    - `--kafka-netflow-consumer=kafka@rb_flow_pre`, Kafka host/topic to listen for netflow\n\n- Configuration\n  - `--rb-config=/opt/rb/etc/f2k/config.json`, File with sensors\n    config (see [Sensor config](#sensor-config))\n\n### Sensors config\nYou need to specify each sensor you want to read netflow from in a JSON file:\n```json\n{\n\t\"sensors_networks\": {\n\t\t\"4.3.2.1\":{\n\t\t\t\"observations_id\": {\n\t\t\t\t\"1\":{\n\t\t\t\t\t\"enrichment\":{\n\t\t\t\t\t\t\"sensor_ip\":\"4.3.2.1\",\n\t\t\t\t\t\t\"sensor_name\":\"flow_test\",\n\t\t\t\t\t\t\"observation_id\":1\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n}\n```\n\nWith this file, you will be listening for netflow coming from\n`4.3.2.1` (this could be a network too, `4.3.2.0/24`), and the JSON output\nwill be sent with that `sensor_ip`, `sensor_name` and `observation_id` keys.\n\n## Others configuration parameters\n\n### Multi-thread\n`--num-threads=N` can be used to specify the number of netflow processing\nthreads.\n\n### Long flow separation\nUse `--separate-long-flows` if you want to divide flow with duration\u003e60s into\nminutes. For example, if the flow duration is 1m30s, f2k will send 1 message\ncontaining 2/3 of bytes and pkts for the minute, and 1/3 of bytes and pkts to\nthe last 30 seconds, like if we had received 2 different flows.\n\n(see [Test 0017](tests/0017-separateLongTimeFlows.c) for more information about\nhow flow are divided)\n\n### Geo information\n`f2k` can add geographic information if you specify\n[Maxmind GeoLite Databases](https://dev.maxmind.com/geoip/legacy/geolite/)\nlocation using:\n  - `--as-list=/opt/rb/share/GeoIP/asn.dat`,\n  - `--country-list=/opt/rb/share/GeoIP/country.dat`,\n\n### Names resolution\nYou can include more flow information, like many object names, with the option\n`--hosts-path=/opt/rb/etc/objects/`. This folder needs to have files with the\nprovided names in order to f2k read them.\n\n### Mac vendor information (`mac_vendor`)\nWith `--mac-vendor-list=mac_vendors` f2k can translate flow source and\ndestination macs, and they will be sending in JSON output as `in_src_mac_name`,\n`out_src_mac_name`, and so on.\n\nThe file `mac_vendors` should be like:\n\n    FCF152|Sony Corporation\n    FCF1CD|OPTEX-FA CO.,LTD.\n    FCF528|ZyXEL Communications Corporation\n\nAnd you can generate it using `make manuf`, that will obtain it automatically\nfrom [IANA Registration Authority](http://standards.ieee.org/develop/regauth/).\n\n### Applications/engine ID (`applications`, `engines`)\n`f2k` can translate applications and engine ID if you specify a list with them,\nlike:\n\n- \\\u003chosts-path\\\u003e/engines\n    ```\n    None            0\n    IANA-L3         1\n    PANA-L3         2\n    IANA-L4         3\n    PANA-L4         4\n    ...\n    ```\n\n- \\\u003chosts-path\\\u003e/applications\n    ```\n    3com-amp3                 50332277\n    3com-tsmux                50331754\n    3pc                       16777250\n    914c/g                    50331859\n    ...\n    ```\n\n### Hosts, domains, vlan (`hosts`, `http_domains`, `vlans`)\nYou can include more information about the flow source and destination (\n`src_name` and `dst_name`) using a hosts list, using the same format as\n`/etc/hosts`. The same can be used with files `vlan`, `domains`, `macs`.\n\n### Netflow probe nets\nYou can specify per netflow probe home nets, so they will be taking into account\nwhen solving client/target IP.\n\nYou could specify them using `home_nets`:\n\n```json\n\"sensors_networks\": { \"4.3.2.0/24\":{ \"2055\":{\n\t\"sensor_name\":\"test1\",\n\t\"sensor_ip\":\"\",\n\t\"home_nets\": [\n\t        {\"network\":\"10.13.30.0/16\", \"network_name\":\"users\" },\n\t        {\"network\":\"2001:0428:ce00:0000:0000:0000:0000:0000/48\",\n\t        \t\t\t\t\"network_name\":\"users6\"}\n\t],\n}}}\n```\n\n### DNS\n`f2k` can make reverse DNS in order to obtain some hosts names. To enable them,\nyou must use:\n- `enable-ptr-dns`, general enable\n- `dns-cache-size-mb`, DNS cache to not repeat PTR queries\n- `dns-cache-timeout-s`, Entry cache timeout\n\n### Template cache\n\n#### Using folder\nYou can specify a folder to save/load templates using\n`--template-cache=/opt/rb/var/f2k/templates`.\n\n#### Using [Apache zookeeper](https://zookeeper.apache.org/)\nIf you want to use zookeeper to share templates between `f2k` instances, you can\nspecify zookeeper host using `--zk-host=127.0.0.1:2181` and a proper timeout to\nread them with `--zk-timeout=30`. Note that you need to compile `f2k` using\n`--enable-zookeeper`.\n\n### [librdkafka](https://github.com/edenhill/librdkafka) options\nAll [librdkafka options](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md).\ncan be used using `-X` (flow producer), `Y` (flow consumer), or `Z`\n(flow discarder) parameter. The argument will be passed directly to librdkafka\nconfig, so you can use whatever config you need.\n\nExample:\n\n```bash\n--kafka-discarder=kafka@rb_flow_discard     # Define host and topic\n--kafka-netflow-consumer=kafka@rb_flow_pre  # Define host and topic\n-X=socket.max.fails=3                       # Define configuration for flow producer\n-X=retry.backoff.ms=100                     # Define configuration for flow producer\n-Y=group.id=f2k                             # Define configuration for consumer\n-Z=group.id=f2k                             # Define configuration for discard producer\n```\n\nRecommended options are:\n\n- `socket.max.fails=3`,\n- `delivery.report.only.error=true`,\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredborder%2Ff2k","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fredborder%2Ff2k","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredborder%2Ff2k/lists"}