{"id":27952299,"url":"https://github.com/uni-tue-kn/tcbee","last_synced_at":"2025-05-08T19:56:47.903Z","repository":{"id":283476701,"uuid":"936024888","full_name":"uni-tue-kn/TCBee","owner":"uni-tue-kn","description":"This repository contains the source code for TCBee, a TCP flow analysis tool recording packet headers and kernel metrics at up to 1.4 Mpps","archived":false,"fork":false,"pushed_at":"2025-05-06T19:45:46.000Z","size":14710,"stargazers_count":8,"open_issues_count":1,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-05-08T19:56:35.975Z","etag":null,"topics":["ebpf","tcp","tcp-analysis","tcp-analyzer","xdp"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/uni-tue-kn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-20T12:07:30.000Z","updated_at":"2025-04-23T08:16:41.000Z","dependencies_parsed_at":null,"dependency_job_id":"c3dfa961-f404-461c-a82f-87b58615822a","html_url":"https://github.com/uni-tue-kn/TCBee","commit_stats":null,"previous_names":["uni-tue-kn/tcbee"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uni-tue-kn%2FTCBee","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uni-tue-kn%2FTCBee/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uni-tue-kn%2FTCBee/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uni-tue-kn%2FTCBee/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/uni-tue-kn","download_url":"https://codeload.github.com/uni-tue-kn/TCBee/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253141467,"owners_count":21860541,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ebpf","tcp","tcp-analysis","tcp-analyzer","xdp"],"created_at":"2025-05-07T17:00:18.108Z","updated_at":"2025-05-08T19:56:47.856Z","avatar_url":"https://github.com/uni-tue-kn.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n \u003cimg src=\"./imgs/tcbee.png\" height=150/\u003e\n\n \u003ch6 style=\"font-size: 10px; margin-left: 200px; margin-top: -30px;\"\u003eBee SVG by \u003ca href=\"https://www.freepik.com/free-vector/cute-bee-insect-animal_136484149.htm#fromView=keyword\u0026page=1\u0026position=2\u0026uuid=42f2e8ed-fa2c-47d9-9793-a1b088c1266d\u0026query=Bees+Svg+File\"\u003eFreepic\u003c/a\u003e \u003c/h6\u003e\n\n \u003ch2\u003eTCBee: A High-Performance and Extensible Tool For TCP Flow Analysis Using eBPF \u003c/h2\u003e\n\n ![image](https://img.shields.io/badge/licence-Apache%202.0-blue) ![image](https://img.shields.io/badge/lang-rust-darkred) ![image](https://img.shields.io/badge/v-0.1.0-yellow) [![TCBee build](https://github.com/uni-tue-kn/TCBee/actions/workflows/tcbee.yml/badge.svg)](https://github.com/uni-tue-kn/TCBee/actions/workflows/tcbee.yml)\n \n\u003c/div\u003e\n\n- [Disclaimer](#disclaimer)\n- [Overview](#overview)\n- [Architecture](#architecture)\n  - [1. Record](#1-record)\n  - [2. Process](#2-process)\n  - [3. Visualize](#3-visualize)\n- [Installation](#installation)\n  - [Prerequisites](#prerequisites)\n  - [Compilation](#compilation)\n- [Working with TCBee](#working-with-tcbee)\n  - [1. Recording Data](#1-recording-data)\n  - [2. Processing Recorded Data](#2-processing-recorded-data)\n  - [3. Visualizing Processed Data](#3-visualizing-processed-data)\n- [Accessing Recorded Data with Custom Scripts](#accessing-recorded-data-with-custom-scripts)\n  - [Using the Rust ts-storage Library](#using-the-rust-ts-storage-library)\n  - [Using Custom Scripts and Programs](#using-custom-scripts-and-programs)\n  - [Accessing the raw data ouput](#accessing-the-raw-data-ouput)\n- [Preview of TCBee](#preview-of-tcbee)\n  - [Recording TCP Flows](#recording-tcp-flows)\n  - [Visualizing CWND Size](#visualizing-cwnd-size)\n  - [Visualizing Multiple Flows](#visualizing-multiple-flows)\n\n## Disclaimer\n\nThis repository contains the first stable version of TCBee and will be improved/refined in the future.\nThe current Todo-List includes\n\n- Documentation for the tools and interfaces\n- Merging tools into a single program\n- Add plugins for the calculation of common TCP congestion metrics\n- Implement InfluxDB interface for faster processing \n- Test and benchmark bottlenecks (eBPF Ringbuf size, File writer, etc.)\n- Cleanup of eBPF and user space code\n- ...\n\nThe current version is tested for linux kernel 6.13.6 and may not work on older or newer kernel versions.\n\n## Overview\n\nThis repository contains the source code for a TCP flow analysis and visualization tool that can monitor any number of TCP flows with up to 1.4 Mpps in total. It uses the Rust programming languages and monitors both packet headers with XDP and TC, and kernel metrics using eBPF.\n\nTCBee\n\n* provides a command-line program to record flows and track current data rates\n* monitors both packet headers for incoming and outgoing packets\n* hooks onto the linux kernel functions `tcp_sendmsg` and `tcp_recvmsg` to read kernel metrics\n* stores recorded data in a structured flow database\n* provides a simple plugin interface to calculate metrics from recorded data and save the results\n* comes with a visualization tool to analyse and compare TCP flow metrics\n* provides a rust library to access flow data for custom visualization tools\n\n\n## Architecture\n\nThe architecture of the TCP analysis tool focuses on achieving a high online processing speed while still being extensible.\nTo that end, the structure of the tool consists of the three phases: **record**, **process**, and **visualize**.\n\n\u003cimg src=\"./imgs/architecture.png\" height=150/\u003e\n\n### 1. Record\n\nThe tool monitors incoming and outgoing TCP traffic, identifies flows and stores all available information in a database.\nFor each flow, the TCP header of every single packet is collected over an eBPF XDP for incoming packets or TC hook for outgoing packets and stored with an associated timestamp.\nFurther, the eBPF tracepoints monitor kernel metrics such as the congestion window size and store them in the same way.\n\n### 2. Process\n\nHere, more complex metrics are extracted such as duplicate ACK events or retransmissions which would otherwise slow down the live recording.\nFurther, TCBee provides a plugin system to define the calculation of new metrics.\nWriting such a plugin uses a simple interfaces and requires no knowledge about the code of TCBee.\n\n### 3. Visualize\n\nThe information from the database can be read by visualization tools that generate graphs or use a GUI to analyze the results.\nTCBee uses a strucutred format with SQLite or InfluxDB databases to simplify access for custom scripts and visualization tools.\n\n## Installation\n\n*Note: TCBee was developed on and is designed for linux systems only. It will not work on MacOS or Windows.*\nThis project was built using the aya rust template: https://github.com/aya-rs/aya-template.\nYou can visit the project for more information on prerequisites and compiling the project for different architectures.\n\n### Prerequisites\n\nTo compile and run the program, the following requirements need to be fulfilled:\n\n- Clang and LLVM (e.g. for Ubuntu `sudo apt install -y llvm clang libelf-dev libclang-dev`)\n- Rustup (\u003e 1.28.1), install via [rustup](https://rustup.rs/)\n- Stable Rust toolchain `rustup toolchain install stable`\n- Nightly Rust toolchain `rustup toolchain install nightly --component rust-src`\n- BPF linker `cargo install bpf-linker`\n\nFor the visualization tool:\n\n- Pkg-config and fontconfig (e.g. for Ubuntu `sudo apt install -y pkg-config fontconfig libfontconfig1-dev`)\n\n### Compilation\n\nYou can build the entire project using `make`, or build single components with `make record`, `make process`, `make viz`.\nThe resulting binaries will be copied into the `install` folder in the root directory.\nThen, move these binaries to any directory that is in your `PATH`.\nThe `tcbee` script is used as main command and runs the other binaries depending on passed arguments.\n\nAlternatively, you can also build the tool parts manually using cargo.\n\n## Working with TCBee\n\nWhen working with TCBee, you can call all sub-programs through the `tcbee` script.\n\n### 1. Recording Data\n\nUse `tcbee record [interface]` to start recording data on the specified interface.\nAvailable options are:\n\n- `-q`, `--quiet` to start the program without the terminal UI\n- `-p`, `--port` to filter for flows that have the specified port as source or destination\n- `--tui-update-ms` to set an alternative update interval of the UI. May help with tearing, default is 100ms.\n\nData recorded by this tool is written as bytes to `*.tcp` files in `/tmp/`.\n\n### 2. Processing Recorded Data\n\nUse `tcbee process` to read the recorded data and generate the flow database.\nCurrently, the flow database will always be created in the same directory as `db.sqlite`.\nIn the future, it will support an InfluxDB backend to speedup processing of large traces.\n\n### 3. Visualizing Processed Data\n\nUse `tcbee viz` to start the visualization tool.\nOnce the tool opens, you can load an `*.sqlite` file to visualize.\nYou can navigate between plotting, multi-flow plotting, processing and settings via the navigation bar.\nThe visualization tool is still in development and you may need to resize the window if fields or buttons are missing.\n\n## Accessing Recorded Data with Custom Scripts\n\nIf you dont want to use the visualization tool, you can access the recorded data directly from the flow database or directly after the recording.\n\n### Using the Rust ts-storage Library\n\n[ts-storage](ts-storage/) contains a database interface created for TCBee.\nIt uses an abstract `TSDBInterface` that provides the same interface independant of the used database systems.\nFor example code and usage, see [ts-storage/README.md](ts-storage/README.md)\n\n### Using Custom Scripts and Programs\n\nYou can generate custom graphs and visualization using your own tools and scripts by accessing the flow database directly.\nTo that end, you either need to implement access over SQLite or InfluxDB depending on the storage format.\nFor a guide on how to read flow data, see [ts-storage/ACCESS.md](ts-storage/ACCESS.md).\n\n### Accessing the raw data ouput\n\nTCBee stores the recorded data in raw byte files under `/tmp/*.tcp`. \nIf you want to read the raw bytes from your own program, take a look at [tcbee-record/tcbee-common/src/bindings/](tcbee-record/tcbee-common/src/bindings/) to find the appropriate structs (struct names that are written end with `_entry`).\n\n## Preview of TCBee\n\n### Recording TCP Flows\n\n\u003cimg alt=\"Recording\" style=\"border-radius: 10px; border: 1px solid #000;\" src=\"imgs/record.png\"/\u003e\n\n\u003cimg alt=\"Recording\" style=\"border-radius: 10px; border: 1px solid #000;\" src=\"imgs/record.webp\"/\u003e\n\n### Visualizing CWND Size\n\n\u003cimg alt=\"TCBee-Viz\" style=\"border-radius: 10px; border: 1px solid #000;\" src=\"imgs/visualize.png\"/\u003e\n\n### Visualizing Multiple Flows\n\n\u003cimg alt=\"TCBee-Viz Multiple Flows\" style=\"border-radius: 10px; border: 1px solid #000;\" src=\"imgs/visualize_multiple_flows.png\"/\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funi-tue-kn%2Ftcbee","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Funi-tue-kn%2Ftcbee","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Funi-tue-kn%2Ftcbee/lists"}