{"id":29356885,"url":"https://github.com/cesnet/ipfixcol2","last_synced_at":"2025-07-09T05:40:44.691Z","repository":{"id":37634337,"uuid":"151271554","full_name":"CESNET/ipfixcol2","owner":"CESNET","description":"High-performance NetFlow v5/v9 and IPFIX collector (RFC7011)","archived":false,"fork":false,"pushed_at":"2025-07-02T13:03:36.000Z","size":2662,"stargazers_count":163,"open_issues_count":26,"forks_count":42,"subscribers_count":20,"default_branch":"master","last_synced_at":"2025-07-02T13:50:58.178Z","etag":null,"topics":["collector","high-performance","ipfix","liberouter","modular","netflow","traffic-monitoring"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CESNET.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-10-02T14:47:04.000Z","updated_at":"2025-07-02T13:03:40.000Z","dependencies_parsed_at":"2024-02-09T14:26:15.603Z","dependency_job_id":"2f651531-4bc4-497f-88e9-ffdc387c0b5b","html_url":"https://github.com/CESNET/ipfixcol2","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/CESNET/ipfixcol2","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CESNET%2Fipfixcol2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CESNET%2Fipfixcol2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CESNET%2Fipfixcol2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CESNET%2Fipfixcol2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CESNET","download_url":"https://codeload.github.com/CESNET/ipfixcol2/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CESNET%2Fipfixcol2/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264402760,"owners_count":23602595,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["collector","high-performance","ipfix","liberouter","modular","netflow","traffic-monitoring"],"created_at":"2025-07-09T05:40:37.522Z","updated_at":"2025-07-09T05:40:44.682Z","avatar_url":"https://github.com/CESNET.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"✨ *Tip: Are you sure your NetFlow/IPFIX probe is working correctly? Be sure with our other project:* |FlowTest|_\n\n-----\n\n.. _FlowTest: https://github.com/CESNET/FlowTest\n.. |FlowTest| replace:: *🌊 FlowTest*\n\n============= =============\nMaster branch |BuildMaster|\n============= =============\n\nIPFIXcol2\n===========\n\nIPFIXcol2 is a flexible, high-performance NetFlow v5/v9 and IPFIX flow data collector designed\nto be extensible by plugins. The second generation of the collector includes many design and\nperformance enhancements compared to the original `IPFIXcol \u003chttps://github.com/CESNET/ipfixcol/\u003e`_.\n\n.. image:: doc/main_img.svg\n\nThe collector allows you to choose combination of input, intermediate and output plugins that\nbest suit your needs. Do you need to receive data over UDP/TCP and store them for long term\npreservation? Or, do you prefer conversion to JSON and processing by other systems?\nNo problem, pick any combination of plugins.\n\n*Features:*\n\n- Input, intermediate and output plugins with various options\n- Parallelized design for high-performance\n- Support for bidirectional flows (biflow)\n- Support for structured data types (i.e. lists)\n- Built-in support for many Enterprise-Specific Information Elements (Cisco, Netscaler, etc.)\n\nAvailable plugins\n-----------------\n\n**Input plugins** - receive NetFlow/IPFIX data. Each can be configured to listen on a specific\nnetwork interface and a port. Multiple instances of these plugins can run concurrently.\n\n- `UDP \u003csrc/plugins/input/udp\u003e`_ - receive NetFlow v5/v9 and IPFIX over UDP\n- `TCP \u003csrc/plugins/input/tcp\u003e`_ - receive IPFIX over TCP\n- `FDS File \u003csrc/plugins/input/fds\u003e`_ - read flow data from FDS File (efficient long-term storage)\n- `IPFIX File \u003csrc/plugins/input/ipfix\u003e`_ - read flow data from IPFIX File\n\n**Intermediate plugins** - modify, enrich and filter flow records.\n\n- `Anonymization \u003csrc/plugins/intermediate/anonymization/\u003e`_ - anonymize IP addresses\n  (in flow records) with Crypto-PAn algorithm\n\n**Output plugins** - store or forward your flows.\n\n- `FDS File \u003csrc/plugins/output/fds\u003e`_ - store all flows in FDS file format (efficient long-term storage)\n- `Forwarder \u003csrc/plugins/output/forwarder\u003e`_ - forward flows as IPFIX to one or mode subcollectors\n- `IPFIX File \u003csrc/plugins/output/ipfix\u003e`_ - store all flows in IPFIX File format\n- `JSON \u003csrc/plugins/output/json\u003e`_ - convert flow records to JSON and send/store them\n- `JSON-Kafka \u003csrc/plugins/output/json-kafka\u003e`_ - convert flow records to JSON and send them to Apache Kafka\n- `Viewer \u003csrc/plugins/output/viewer\u003e`_ - convert IPFIX into plain text and print\n  it on standard output\n- `Time Check \u003csrc/plugins/output/timecheck\u003e`_ - flow timestamp check\n- `Dummy \u003csrc/plugins/output/dummy\u003e`_ - simple output module example\n- `Overview \u003csrc/plugins/output/overview\u003e`_ - get a quick overview of the IPFIX\n  fields the collector is receiving from the probe\n- `lnfstore \u003cextra_plugins/output/lnfstore\u003e`_ (*) - store all flows in nfdump compatible\n  format for long-term preservation\n- `UniRec \u003cextra_plugins/output/unirec\u003e`_ (*)  - send flow records in UniRec format\n  via TRAP communication interface (into Nemea modules)\n- `ClickHouse \u003cextra_plugins/output/clickhouse\u003e`_ (*)  - insert flow records\n  into a ClickHouse database\n\n\\* Must be installed individually due to extra dependencies\n\nHow to install\n--------------\n\nIf you are running a RHEL system or one of its derivatives, the easiest way to\nget IPFIXcol installed is using our Copr package repository.\n\n.. code-block:: bash\n\n    $ dnf install 'dnf-command(copr)'  # Extra step necessary on some systems\n    $ dnf copr enable @CESNET/IPFIXcol\n    $ dnf install ipfixcol2\n\nFor other systems, follow the build instructions below.\n\nHow to build\n------------\n\nIPFIXcol is based on `libfds \u003chttps://github.com/CESNET/libfds/\u003e`_ library that provides\nfunctions for IPFIX parsing and manipulation. First of all, install the library.\nFor more information visit the project website and follow installation instructions.\n\nHowever, you have to typically do following steps: (extra dependencies may be required)\n\n.. code-block:: bash\n\n    $ git clone https://github.com/CESNET/libfds.git\n    $ cd libfds\n    $ mkdir build \u0026\u0026 cd build \u0026\u0026 cmake .. -DCMAKE_INSTALL_PREFIX=/usr\n    $ make\n    # make install\n\nSecond, install build dependencies of the collector\n\n**RHEL/CentOS:**\n\n.. code-block::\n\n    yum install gcc gcc-c++ cmake make python3-docutils zlib-devel librdkafka-devel\n    # Optionally: doxygen pkgconfig\n\n* Note: latest systems (e.g. Fedora/CentOS Stream 8) use ``dnf`` instead of ``yum``.\n* Note: package ``python3-docutils`` may by also named as ``python-docutils`` or ``python2-docutils``\n* Note: package ``pkgconfig`` may by also named as ``pkg-config``\n* Note: CentOS Stream 8 usually requires additional system repositories to be enabled:\n\n.. code-block::\n\n    dnf -y install epel-release\n    dnf config-manager --set-enabled appstream powertools\n\n* Note: Oracle Linux 8 usually requires additional system repositories to be enabled:\n\n.. code-block::\n\n    dnf -y install oracle-epel-release-el8\n    dnf config-manager --set-enabled ol8_appstream ol8_codeready_builder\n\n**Debian/Ubuntu:**\n\n.. code-block::\n\n    apt-get install gcc g++ cmake make python3-docutils zlib1g-dev librdkafka-dev\n    # Optionally: doxygen pkg-config\n\nFinally, build and install the collector:\n\n.. code-block:: bash\n\n    $ git clone https://github.com/CESNET/ipfixcol2.git\n    $ cd ipfixcol2\n    $ mkdir build \u0026\u0026 cd build \u0026\u0026 cmake ..\n    $ make\n    # make install\n\nHow to configure and start IPFIXcol\n-----------------------------------\n\nBefore you can start IPFIXcol, you have to prepare a configuration file. The file describes how\nIPFIXcol is configured at startup, which plugins are used and, for example, where flow data will\nbe stored. The structure of the configuration is described\n`here \u003cdoc/sphinx/configuration.rst\u003e`_. Several configuration examples that demonstrate features\nof the collector are given in the section\n\"`Example configuration files \u003cdoc/sphinx/configuration.rst#example-configuration-files\u003e`_\".\n\nFAQ\n--------------\n\nDo you have any troubles? Unable to build and run the collector? *Feel free to submit a new issue.*\n\nWe are open to new ideas! For example, are you missing a specific plugin that could\nbe useful also for other users? Please, share your experiences and thoughts.\n\n----\n\n:Q: My exporter sends flow data over UDP, however, the IPFIXcol doesn't process/store any data\n    immediately after start.\n:A: This is normal behaviour caused by UDP transport protocol. It may take up few minutes until\n    the first record is processed based on template refresh interval on the exporter.\n    For more information, see documentation of `UDP \u003csrc/plugins/input/udp\u003e`_ plugin.\n\n:Q: The collector is not able to find a plugin. What should I do?\n:A: First of all, make sure that the plugin is installed. Some plugins (e.g. Unirec) are optional\n    and must be installed separately. Therefore, list all available plugins\n    using ``ipfixcol2 -L`` and check if the plugin is on the list. If not, see the plugin page\n    for help. If the problem still persists, check if the plugin is installed in the correct\n    directory. Since plugins might be placed in different locations on different platforms,\n    show help using ``ipfixcol2 -h`` and see the default value of ``-p PATH`` parameter.\n    In some situations, it is also possible that the plugin cannot be loaded (even when\n    it is properly installed) due to additional dependencies (e.g. missing library etc.).\n    If this is the issue, use ``ipfixcol2 -L -v`` and there might be a message like this\n    ``WARNING: Configurator (plugin manager): Failed to open file... (some reason)``\n    on the first line that might help you.\n\n:Q: How can I add more IPFIX fields into records?\n:A: The collector receives flow records captured and prepared by an exporter. IPFIX is an\n    unidirectional protocol which means that the collector is not able to instruct the exporter\n    what to measure or how to behave. If you want to enhance your records, please, check\n    configuration of your exporter.\n\n:Q: After *manual build and installation* the collector is unable to start and a message similar to\n    ``error while loading shared libraries: libfds.so.0: cannot open shared object file: No such file or directory``\n    is given.\n:A: Make sure that ``libfds`` is installed properly and your system is able to locate it.\n    Some systems (e.g. RHEL/CentOS/Fedora) for historical reason doesn't search for shared libraries\n    in the default installation directory where the ``libfds`` is installed. You can permanently\n    include this directory. For example, if the library is located in ``/usr/local/lib64``, use\n    as administrator \"``echo \"/usr/local/lib64\" \u003e /etc/ld.so.conf.d/local64.conf \u0026\u0026 ldconfig``\"\n    or temporarily change an environment variable\n    \"``export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib64/``\"\n\n.. |BuildMaster| image:: https://github.com/CESNET/ipfixcol2/actions/workflows/main.yml/badge.svg?branch=master\n   :target: https://github.com/CESNET/ipfixcol2/tree/master\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcesnet%2Fipfixcol2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcesnet%2Fipfixcol2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcesnet%2Fipfixcol2/lists"}