{"id":22663698,"url":"https://github.com/yuuki/shawk","last_synced_at":"2025-08-20T20:33:08.305Z","repository":{"id":40397093,"uuid":"191506750","full_name":"yuuki/shawk","owner":"yuuki","description":"[PoC]  A socket-based tracing system for discovering network service dependencies. (renamed from transtracer) ","archived":false,"fork":false,"pushed_at":"2024-12-04T01:00:54.000Z","size":6112,"stargazers_count":52,"open_issues_count":12,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-12-04T02:18:44.299Z","etag":null,"topics":["dependency","distributed-tracing","monitoring","network-monitoring","observability","tcp","tracing","transport"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yuuki.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-06-12T05:56:43.000Z","updated_at":"2024-07-31T17:13:00.000Z","dependencies_parsed_at":"2023-12-18T20:29:53.346Z","dependency_job_id":"121e2a40-a8de-470f-aa44-32ce6196900c","html_url":"https://github.com/yuuki/shawk","commit_stats":null,"previous_names":["yuuki/transtracer"],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuuki%2Fshawk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuuki%2Fshawk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuuki%2Fshawk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yuuki%2Fshawk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yuuki","download_url":"https://codeload.github.com/yuuki/shawk/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230454430,"owners_count":18228392,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dependency","distributed-tracing","monitoring","network-monitoring","observability","tcp","tracing","transport"],"created_at":"2024-12-09T12:32:41.166Z","updated_at":"2024-12-19T15:09:56.973Z","avatar_url":"https://github.com/yuuki.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Shawk\n\n[![GitHub Actions status](https://github.com/yuuki/shawk/workflows/Test/badge.svg)](https://github.com/yuuki/shawk/actions)\n[![Latest Version](http://img.shields.io/github/release/yuuki/shawk.svg?style=flat-square)](https://github.com/yuuki/shawk/releases)\n[![Go Report Card](https://goreportcard.com/badge/github.com/yuuki/shawk)](https://goreportcard.com/report/github.com/yuuki/shawk)\n[![License](http://img.shields.io/:license-mit-blue.svg)](http://doge.mit-license.org)\n\n\u003cimg alt=\"shawk-logo\" src=\"https://github.com/yuuki/shawk/raw/master/doc/images/logo.png\" width=\"200\"\u003e\n\n----\n\nShawk is a socket-based tracing infrastructure for discovering network dependencies among processes in distributed applications. Shawk has an architecture of monitoring network sockets, which are endpoints of TCP connections, to trace the dependencies.\n\n## Contributions\n\n- As long as applications use the TCP protocol stack in the Linux kernel, the dependencies are discovered by Transtracer.\n- The monitoring does not affect the network delay of the applications because the processing that only reads the connection information from network sockets is independent of the application communication.\n\n## System Overview\n\n![System structure](/doc/images/system_structure.png \"System structure\")\n\nThis figure shows the system configuration for matching the connection information related to multiple hosts and for creating a dependency graph. Tracer running on each host sends connection information to the central Connection Management DataBase (CMDB).\n\n![Socket diagnosis in polling mode](/doc/images/socket_diagnosis.png \"Socket diagnosis in polling mode\")\n\nThis figure shows how to retrieve socket information for TCP connections. When the Tracer process runs on the host, the Tracer process queries the Linux kernel and obtains a snapshot of the active TCP connection status from the socket corresponding to each connection. At the same time, the Tracer process acquires the process information corresponding to each connection. Then it links each connection and each process.\n\n## Getting Started\n\nSee [Quickstart document](./doc/quickstart.md).\n\n## Requirements\n\n- OS: Linux\n- RDBMS: PostgreSQL 10+\n\n## Usage\n\n```shell-session\n$ shawk --help\nUsage: shawk [options]\n\n  A socket-based tracing system for discovering network dependencies in distributed applications.\n\nCommands:\n  look           show dependencies starting from a specified node.\n  probe          start agent for collecting flows and processes.\n  create-scheme  create CMDB scheme.\n\nOptions:\n  --version         print version\n  --credits         print credits\n  --help, -h        print help\n```\n\n### shawk probe\n\nRun a daemon process of scanning connections in polling mode (default).\n\n```shell-session\n# SHAWK_PROBE_MODE=polling SHAWK_PROBE_INTERVAL=1s SHAWK_FLUSH_INTERVAL=10s SHAWK_CMDB_URL=postgres://shawk:password@127.0.0.1:5432/shawk?sslmode=disable\u0026connect_timeout=1 shawk probe\n```\n\nRun a daemon process in streaming mode, which internaly uses eBPF.\n\n```shell-session\n# SHAWK_PROBE_MODE=streaming SHAWK_PROBE_INTERVAL=1s SHAWK_CMDB_URL=postgres://shawk:password@127.0.0.1:5432/shawk?sslmode=disable\u0026connect_timeout=1 shawk probe\n```\n\nRun scanning connections only once.\n\n```shell-session\n# SHAWK_PROBE_MODE=streaming SHAWK_PROBE_INTERVAL=1s SHAWK_CMDB_URL=postgres://shawk:password@127.0.0.1:5432/shawk?sslmode=disable shawk probe --once\n```\n\n### shawk look\n\n```shell-session\n# SHAWK_CMDB_URL=postgres://shawk:password@127.0.0.1:5432/shawk?sslmode=disable shawk look --ipv4 10.0.0.10\n10.0.0.10:80 (’nginx’, pgid=4656)\n└\u003c-- 10.0.0.11:many (’wrk’, pgid=5982) 10.0.0.10:80 (’nginx’, pgid=4656)\n└--\u003e 10.0.0.12:8080 (’python’, pgid=6111) 10.0.0.10:many (’fluentd’, pgid=2127)\n└--\u003e 10.0.0.13:24224 (’fluentd’, pgid=2001)\n```\n\n## Papers (including proceedings)\n\n1. Yuuki Tsubouchi, Masahiro Furukawa, Ryosoke Matsumoto, Transtracer: Automatically Tracing for Processes Dependencies in Distributed Systems by Monitoring Endpoints of TCP/UDP, IPSJ Internet and Operation Technology Symposium (IOTS2019), Vol. 2019, pp. 64-71, 2019. [[paper](https://yuuk.io/papers/shawk_iots2019.pdf)] [[slide](https://speakerdeck.com/yuukit/udptong-xin-falsezhong-duan-dian-falsejian-shi-niyoruhurosesujian-yi-cun-guan-xi-falsezi-dong-zhui-ji-8bc9ca63-0751-40fd-9ad5-2f1ea692b9b0)]\n\n## License\n\n[MIT](LICENSE)\n\n## Author\n\n[yuuki](https://github.com/yuuki)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyuuki%2Fshawk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyuuki%2Fshawk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyuuki%2Fshawk/lists"}