{"id":20840242,"url":"https://github.com/stefandeveloper/heifip","last_synced_at":"2025-05-08T21:45:18.798Z","repository":{"id":54528115,"uuid":"522472839","full_name":"stefanDeveloper/heiFIP","owner":"stefanDeveloper","description":"heiFIP: A tool to convert network traffic into images for ML use cases","archived":false,"fork":false,"pushed_at":"2024-08-30T16:02:46.000Z","size":26310,"stargazers_count":16,"open_issues_count":2,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-09-19T14:38:32.506Z","etag":null,"topics":["cybersecurity","dataset-generation","image-generator","machine-learning","netflow","network-classification","packet-analyser","pcap","pcap-parser","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"eupl-1.2","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stefanDeveloper.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-08T08:51:33.000Z","updated_at":"2024-09-18T07:50:02.000Z","dependencies_parsed_at":"2024-03-21T14:05:08.422Z","dependency_job_id":null,"html_url":"https://github.com/stefanDeveloper/heiFIP","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanDeveloper%2FheiFIP","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanDeveloper%2FheiFIP/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanDeveloper%2FheiFIP/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stefanDeveloper%2FheiFIP/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stefanDeveloper","download_url":"https://codeload.github.com/stefanDeveloper/heiFIP/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225110389,"owners_count":17422413,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cybersecurity","dataset-generation","image-generator","machine-learning","netflow","network-classification","packet-analyser","pcap","pcap-parser","python"],"created_at":"2024-11-18T01:15:42.693Z","updated_at":"2024-11-18T01:15:43.355Z","avatar_url":"https://github.com/stefanDeveloper.png","language":"Python","readme":"![heiFIP Logo](https://raw.githubusercontent.com/stefanDeveloper/heiFIP/main/assets/heiFIP_logo.png?raw=true)\n\n\n--------------------------------------------------------------------------------\n\n**heiFIP** stands for Heidelberg Flow Image Processor.\nIt is a tool designed to extract essential parts of packets and convert them into images for deep learning purposes.\nheiFIP supports different formats and orientations.\nCurrently, we only support **offline** network data analysis.\nHowever, we plan to adapt our library to support **online** network data too to enable live-probing of models.\n\n\u003ctable\u003e\n\u003ctr\u003e\n  \u003ctd\u003e\u003cb\u003eLive Notebook\u003c/b\u003e\u003c/td\u003e\n  \u003ctd\u003e\n    \u003ca href=\"https://mybinder.org/v2/gh/stefanDeveloper/heiFIP-tutorials/HEAD?labpath=demo_notebook.ipynb\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/notebook-launch-blue?logo=jupyter\u0026style=for-the-badge\" alt=\"live notebook\" /\u003e\n    \u003c/a\u003e\n  \u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd\u003e\u003cb\u003eLatest Release\u003c/b\u003e\u003c/td\u003e\n  \u003ctd\u003e\n    \u003ca href=\"https://pypi.python.org/pypi/heifip\"\u003e\n    \u003cimg src=\"https://img.shields.io/pypi/v/heifip.svg?logo=pypi\u0026style=for-the-badge\" alt=\"latest release\" /\u003e\n    \u003c/a\u003e\n  \u003c/td\u003e\n\u003c/tr\u003e\n\n\u003ctr\u003e\n  \u003ctd\u003e\u003cb\u003eSupported Versions\u003c/b\u003e\u003c/td\u003e\n  \u003ctd\u003e\n    \u003ca href=\"https://pypi.org/project/heifip/\"\u003e\n    \u003cimg src=\"https://img.shields.io/pypi/pyversions/heifip?logo=python\u0026style=for-the-badge\" alt=\"python3\" /\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://pypi.org/project/heifip/\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/pypy-3.7%20%7C%203.8%20%7C%203.9-blue?logo=pypy\u0026style=for-the-badge\" alt=\"pypy3\" /\u003e\n    \u003c/a\u003e\n  \u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd\u003e\u003cb\u003eProject License\u003c/b\u003e\u003c/td\u003e\n  \u003ctd\u003e\n    \u003ca href=\"https://github.com/stefanDeveloper/heifip/blob/main/LICENSE\"\u003e\n    \u003cimg src=\"https://img.shields.io/pypi/l/heifip?logo=gnu\u0026style=for-the-badge\u0026color=blue\" alt=\"License\" /\u003e\n    \u003c/a\u003e\n  \u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd\u003e\u003cb\u003eCitation\u003c/b\u003e\u003c/td\u003e\n  \u003ctd\u003e\n    \u003ca href=\"https://zenodo.org/badge/latestdoi/522472839\"\u003e\n    \u003cimg src=\"https://zenodo.org/badge/522472839.svg?style=for-the-badge\" alt=\"Citation\" /\u003e\n    \u003c/a\u003e\n  \u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd\u003e\u003cb\u003eContinuous Integration\u003c/b\u003e\u003c/td\u003e\n  \u003ctd\u003e\n    \u003ca href=\"https://github.com/stefanDeveloper/heifip/actions/workflows/build_test_linux.yml\"\u003e\n    \u003cimg src=\"https://img.shields.io/github/actions/workflow/status/stefanDeveloper/heifip/build_test_linux.yml?branch=main\u0026logo=linux\u0026style=for-the-badge\u0026label=linux\" alt=\"Linux WorkFlows\" /\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/stefanDeveloper/heifip/actions/workflows/build_test_macos.yml\"\u003e\n    \u003cimg src=\"https://img.shields.io/github/actions/workflow/status/stefanDeveloper/heifip/build_test_macos.yml?branch=main\u0026logo=apple\u0026style=for-the-badge\u0026label=macos\" alt=\"MacOS WorkFlows\" /\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://github.com/stefanDeveloper/heifip/actions/workflows/build_test_windows.yml\"\u003e\n    \u003cimg src=\"https://img.shields.io/github/actions/workflow/status/stefanDeveloper/heifip/build_test_windows.yml?branch=main\u0026logo=windows\u0026style=for-the-badge\u0026label=windows\" alt=\"Windows WorkFlows\" /\u003e\n    \u003c/a\u003e\n  \u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n## Table of Contents\n\n- [**Main Features**](#main-features)\n- [**Motivation**](#motivation)\n- [**Examples**](#examples)\n- [**Getting Started**](#getting-started)\n  - [**Building from source**](#building-from-source)\n- [**Citation**](#citation)\n  - [**Credits**](#credits)\n  - [**Authors**](#authors)\n- [**License**](#license)\n\n## Motivation\n\nThe idea to create heiFIP came from working with Deep Learning approaches to classify malware traffic on images. Many papers use image representation of network traffic, but reproducing their results was quite cumbersome. As a result, we found that there is currently no official library that supports reproducible images of network traffic. For this reason, we developed heiFIP to easily create images of network traffic and reproduce ML/DL results. Researchers can use this library as a baseline for their work to enable other researchers to easily recreate their findings.\n\n## Main Features\n\n- **Different Images**: Currently, we support plain packet to byte representation, and flow to byte representation with one channel each. An image is created with same width and height for a quadratic representation.\n  - **Flow Images** converts a set of packets into an image. It supports the following modifications:\n    - **Max images dimension** allows you to specify the maximum image dimension. If the packet is larger than the specified size, it will cut the remaining pixel.\n    - **Min image dimesion** allows you to specify the minimum image dimension. If the packet is smaller than the specified size, it fills the remaining pixel with 0.\n    - **Remove duplicates** allows you to automatically remove same traffic.\n    - **Append** each flow to each other or write each packet to a new row.\n    - **Tiled** each flow is tiled into a square image representation.\n    - **Min packets per flow** allows you to specify the minimum number of packets per flow. If the total number of packets is too small, no image will be created.\n    - **Max packets per flow** allows you to specify the maximum number of packets per flow. If the total number of packets is too great, the remaining images are discarded.\n  - **Packet Image** converts a single packet into an image.\n  - **Markov Transition Matrix Image**: converts a packet or a flow into a Markov representation.\n- **Header** processing allows you to customize header fields of different protocols. It aims to remove biasing fields. For more details look into [header.py](https://github.com/stefanDeveloper/heiFIP/blob/main/heifip/plugins/header.py)\n- **Remove Payload** options allows you to only work on header data.\n- **Fast and flexible**: We rely on [Scapy](https://github.com/secdev/scapy) for our sniffing and header processing. Image preparation is based on raw bytes.\n- **Machine learning orientation**: heiFIP aims to make Deep Learning approaches using network data as images reproducible and deployable. Using heiFIP as a common framework enables researches to test and verify their models.\n\n## Examples\n\n| Image Type | Description | Example |\n|------------|-------------|---------|\n| Packet | Converts a single packet into a square image. Size depends on the total length | ![SMB Connection](https://raw.githubusercontent.com/stefanDeveloper/heiFIP/main/examples/packet.png?raw=true) |\n| Flow | Converts a flow packet into a square image | ![SMB Connection](https://raw.githubusercontent.com/stefanDeveloper/heiFIP/main/examples/flow-tiled.png?raw=true) |\n| Markov Transition Matrix Packet | Converts a packet into a Markov Transition Matrix. Size is fixed to 16x16. | ![SMB Connection](https://raw.githubusercontent.com/stefanDeveloper/heiFIP/main/examples/markov-packet.png?raw=true) |\n| Markov Transition Matrix Flow | Converts a flow into a Markov Transition Matrix. It squares the image based on the number of packets | ![SMB Connection](https://raw.githubusercontent.com/stefanDeveloper/heiFIP/main/examples/markov-flow.png?raw=true) |\n\n## Getting Started\n\nInstall our package using PyPi\n\n```sh\npip install heifip\n```\nNow, you can use the integrate CLI:\n\n```sh\n\u003e fip\nUsage: fip [OPTIONS] COMMAND [ARGS]...\n\nOptions:\n  --version   Show the version and exit.\n  -h, --help  Show this message and exit.\n\nCommands:\n  extract\n```\n\nTo extract images from PCAPs, we currently split the command into flow and packet:\n\n```sh\n\u003e fip extract\nStarting FlowImageProcessor CLI\nUsage: fip extract [OPTIONS] COMMAND [ARGS]...\n\nOptions:\n  -h, --help  Show this message and exit.\n\nCommands:\n  flow\n  packet\n\n# Show help information\n\u003e fip extract [flow/packet]-h\nStarting FlowImageProcessor CLI\nUsage: fip extract flow [OPTIONS]\n\nOptions:\n  -w, --write PATH            Destination file path, stores result  [required]\n  -r, --read PATH             [required]\n  -t, --threads INTEGER       Number of parallel threads that can be used\n                              [default: 4]\n  --preprocess [NONE|HEADER]  Applies a preprocessing to the input data: none:\n                              No preprocessing payload: Only payload data is\n                              used header: Preprocesses headers\n                              (DNS,HTTP,IP,IPv6,TCP,UDP supported) to remove\n                              some biasing data  [default: NONE]\n  --min_im_dim INTEGER        Minimum dim ouput images need to have, 0=No\n                              minimum dim  [default: 0]\n  --max_im_dim INTEGER        Maximum dim ouput images can have, 0=No maximum\n                              dim  [default: 0]\n  --remove_duplicates         Within a single output folder belonging to a\n                              single input folder no duplicate images will be\n                              produced if two inputs lead to the same image\n  --min_packets INTEGER       Minimum packets that a FlowImage needs to have,\n                              0=No minimum packets per flow  [default: 0]\n  --max_packets INTEGER       Minimum packets that a FlowImage needs to have,\n                              0=No minimum packets per flow  [default: 0]\n  --append\n  --tiled\n  --width INTEGER             [default: 128]\n  -h, --help                  Show this message and exit.\n\n\u003e fip extract flow -r /PATH/PCAPs -w /PATH/IMAGES\n```\n\nImport FIPExtractor to run it inside your program:\n\n```python\nextractor = FIPExtractor()\nimg = extractor.create_image('./test/pcaps/dns/dns-binds.pcap')\nextractor.save_image(img, './test/pcaps/dns/dns-binds.pcap')\n```\n\n### Building from source\n\nSimply run:\n\n```\npip install .\n```\n\n### Publications that use heiFIP\n\n- [A Generalizable Approach for Network Flow Image Representation for Deep Learning] - CSNet 23\n- [Explainable artificial intelligence for improving a session-based malware traffic classification with deep learning] - SSCI 23\n\n\n## Credits\n\n[NFStream](https://github.com/nfstream/nfstream) for the inspiration of the `README.md` and workflow testing.\n\n### Authors\n\nThe following people contributed to heiFIP:\n\n- [Stefan Machmeier](https://github.com/stefanDeveloper): Creator\n- [Manuel Trageser](https://github.com/maxi99manuel99): Header extraction and customization.\n\n## License\n\nThis project is licensed under the  EUPL-1.2 [**License**](license) - see the License file for details\n\n[license]: https://github.com/stefanDeveloper/heiFIP/blob/main/LICENSE","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstefandeveloper%2Fheifip","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstefandeveloper%2Fheifip","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstefandeveloper%2Fheifip/lists"}