{"id":13581440,"url":"https://github.com/m-lab/tcp-info","last_synced_at":"2025-08-16T12:33:30.640Z","repository":{"id":34395076,"uuid":"126595257","full_name":"m-lab/tcp-info","owner":"m-lab","description":"Fast tcp-info collector in Go","archived":false,"fork":false,"pushed_at":"2024-06-10T05:08:35.000Z","size":565,"stargazers_count":43,"open_issues_count":29,"forks_count":8,"subscribers_count":11,"default_branch":"main","last_synced_at":"2024-10-29T14:46:53.957Z","etag":null,"topics":["platform","tcp-info"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/m-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-03-24T12:03:56.000Z","updated_at":"2024-09-25T14:21:55.000Z","dependencies_parsed_at":"2024-01-16T20:33:31.127Z","dependency_job_id":"fdfc40a0-a303-4e7d-95c6-425f95e7d108","html_url":"https://github.com/m-lab/tcp-info","commit_stats":{"total_commits":197,"total_committers":11,"mean_commits":17.90909090909091,"dds":0.4517766497461929,"last_synced_commit":"d72b41f1c48272089fc40c20b3dfafce07339ea3"},"previous_names":[],"tags_count":28,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Ftcp-info","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Ftcp-info/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Ftcp-info/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Ftcp-info/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/m-lab","download_url":"https://codeload.github.com/m-lab/tcp-info/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229671022,"owners_count":18104986,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["platform","tcp-info"],"created_at":"2024-08-01T15:02:01.768Z","updated_at":"2024-12-16T23:09:46.157Z","avatar_url":"https://github.com/m-lab.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# tcp-info\n\n[![GoDoc](https://godoc.org/github.com/m-lab/tcp-info?status.svg)](https://godoc.org/github.com/m-lab/tcp-info) [![Build Status](https://travis-ci.org/m-lab/tcp-info.svg?branch=master)](https://travis-ci.org/m-lab/tcp-info) [![Go Report Card](https://goreportcard.com/badge/github.com/m-lab/tcp-info)](https://goreportcard.com/report/github.com/m-lab/tcp-info) [![Coverage Status](https://coveralls.io/repos/m-lab/tcp-info/badge.svg?branch=master)](https://coveralls.io/github/m-lab/tcp-info?branch=master)\n\nThe `tcp-info` tool executes a polling loop that tracks the measurement statistics of every open TCP socket on a system.  Data is written, in `JSONL` format (refered to internally as *ArchivedRecord*), to files compressed using `zstd`.  This tool forms the basis of a lot of measurements on the Kubernetes-based [Measurement Lab](https://measurementlab.net) platform.\n\nWe expect most people will run this tool using a\ndocker container.  To invoke, with data written to ~/data, and prometheus\nmetrics published on port 7070:\n\n```bash\ndocker run --network=host -v ~/data:/home/ -it measurementlab/tcp-info -prom=7070\n```\n\n## Fast tcp-info collector in Go\n\nThis repository uses the netlink API to collect inet_diag messages, partially parses them, and caches the intermediate representation.\nIt then detects differences from one scan to the next, and queues connections that have changed for logging.\nIt logs the intermediate representation through external zstd processes to one file per connection.\n\nThe previous version uses protobufs, but we have discontinued that largely because of the increased maintenance overhead, and risk of losing unparsed data.\nInstead, we are now using *ArchivedRecord* which is partially parsed netlink messages, mostly in base64 encoded blobs, marshaled to JSONL format, with one JSON object per line.\n\nTo run the tests or the collection tool, you will also require zstd, which can be installed with:\n\n```bash\nbash \u003c(curl -fsSL https://raw.githubusercontent.com/horta/zstd.install/master/install)\n```\n\nOR\n\n```bash\nsudo apt-get update \u0026\u0026 sudo apt-get install -y zstd\n```\n\n## Example sidecar\n\nThe tcp-info eventsocket interface allows sidecar services to receive \"open\" and\n\"close\" events on a unix domain socket connection. A simple reference\nimplementation `cmd/example-eventsocket-client` can be started using\n`docker-compose`.\n\n```bash\ndocker-compose up\n```\n\nNew TCP events are processed by the `example-eventsocket-client` sidecar and\nlogged to stderr. You may trigger a TCP connection from within the TCPINFO\ncontainer using a command like:\n\n```bash\ndocker exec -it tcp-info_tcpinfo_1 wget www.google.com\n```\n\n## Parse library and command line tools\n\n### CSV tool\n\nThe cmd/csvtool directory contains a tool for parsing ArchivedRecord and producing CSV files.  Currently reads netlink-jSONL from stdin and writes CSV to stdout.\n\n## Code Layout\n\n* inetdiag - code related to include/uapi/linux/inet_diag.h.  All structs will be in structs.go\n* tcp - Should include ONLY the code related to include/uapi/linux/tcp.h\n* parse - code related to parsing the messages in inetdiag and tcp.\n* zstd - zstd reader and writer.\n* saver - code related to writing ParsedMessages to files.\n* cache - code to cache netlink messages and detect changes.\n* collector - code related to collecting netlink messages from the kernel.\n\n### Dependencies (as of March 2019)\n\n* saver: inetdiag, cache, parse, tcp, zstd\n* collector: parse, saver, inetdiag, tcp\n* main.go: collector, saver, parse (just for sanity check)\n* cache: parse\n* parse: inetdiag\n\nAnd (almost) all package use metrics.\n\n#### Layers for main.go (each layer depends only on items to right, or lower layers)\n\n1. main.go\n1. collector \u003e saver \u003e cache\n1. netlink \u003e inetdiag\n1. tcp, zstd, metrics\n\n#### Layers for parse package\n\n1. parse (used by command line tools, etl)\n1. netlink \u003e inetdiag\n1. tcp, zstd, metrics\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Ftcp-info","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fm-lab%2Ftcp-info","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Ftcp-info/lists"}