{"id":13775025,"url":"https://github.com/yandex/tcplanz","last_synced_at":"2025-04-12T04:30:35.891Z","repository":{"id":75743166,"uuid":"44682927","full_name":"yandex/tcplanz","owner":"yandex","description":"TCPDump latency analyzer","archived":false,"fork":false,"pushed_at":"2017-12-24T00:28:33.000Z","size":28,"stargazers_count":74,"open_issues_count":2,"forks_count":16,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-26T00:01:45.047Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yandex.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS"}},"created_at":"2015-10-21T14:43:21.000Z","updated_at":"2025-01-30T07:29:01.000Z","dependencies_parsed_at":"2023-06-07T13:15:30.858Z","dependency_job_id":null,"html_url":"https://github.com/yandex/tcplanz","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yandex%2Ftcplanz","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yandex%2Ftcplanz/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yandex%2Ftcplanz/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yandex%2Ftcplanz/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yandex","download_url":"https://codeload.github.com/yandex/tcplanz/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248516975,"owners_count":21117360,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T17:01:32.930Z","updated_at":"2025-04-12T04:30:35.867Z","avatar_url":"https://github.com/yandex.png","language":"Python","funding_links":[],"categories":["\u003ca id=\"b293f791ec9366957733415323755aa6\"\u003e\u003c/a\u003eTcpdump"],"sub_categories":[],"readme":"#### What is this for ####\nThis project allows to get web server performance metric from traffic TCPDump. \nThis allows to measure up metric like **html delivery time** which are usually not available in runtime and have second opinion about other metrics like **server latency**.\n\n\n#### Installation ####\nThis is not meant to be used directy on frontend, because it could consume lot of memory and CPU parsing pcap files.\nPlease install on development server or workstation\n\nBefore installation you need python (pypy is preferrable because it faster on task like this).\nOnly other dependance is dpkt (python tcpdump parsing library) see https://dpkt.readthedocs.org\nIf you use pip to install dpkt please say **pypy -m pip install dpkt** instead of **pip install dpkt**,\nsecond spell installs dpkt for python not pypy.\n\n#### How to get data ####\n\nYou should dump on frontend/reverse proxy/balancer i.e. something which holds tcp connections directly to users\n\nSimple way:\n   **sudo tcpdump -w data.pcap port 80**\n\nProduction way:\n   * use nice -10 to give additional priority to tcpdump\n   * use -B option for largest buffer available on your system. Sometimes tcpdump drop some packets because disk is busy\n   * use -C option to split file by files of reasonable size (1-10Gb) this is easier for copying/etc\n   * use -z to compress splitted files after writing, note: nice/ionice gzip to prevent it from preemping tcpdump \n   * dump only traffic you need, i.e. use filters \"port 80 or port 8080\"\n   * if you debugging frontend itself consider dumping both traffic to user and backend. You may find something interesting comparing timings.\n\n\n#### Parsing ####\n```\n   decode-pcap.py \u003coutdir\u003e (parse|split|sparse) \u003cinput files shoild have extensions .pcap or .pcap.gz\u003e \n\n   outdir - directory where http.txt and debug.txt will be created.\n\n   parse  - just parse files. will use lot of memory because every active session is kept in memory until closed. \n            and tcpdump has a lot of sessions which never ends. Session which crossed tcpdump files will be preserved.\n            order of files is important, i.e. better to have timestamp or number inside file name.\n   split  - extract tcp sessions from pcap files split it to several new files. Each session will be only in one file.\n            order of packets will be changed. I.e. session 1 - packets 1..N, session 2 packets N+1..M, etc.\n            timestamps will be preserved. \n   sparse - parse splited files. Actually it is same as just run programs on all files one by one.\n            sessions across files will be separated. \n```\n\nExamples: \n\n```\n     decode-pcap.py out parse tcpdump.pcap.gz \n     (test example)\n\n     decode-pcap.py splitted split 100Gb-dump/*.pcap.gz\n     decode-pcap.py out sparse splitted/*.pcap.gz \n\n     (real world example. If you want to parse 100Gb of tcpdump you may need comparable amount of memory for split \n     operation. Comparable means 100Gb if all http sessions is keep-alive and not finished inside file. And small amount of memory if all sessions are short. \n     In real world you have mixed traffic so be ready to have 20Gb-30Gb of memory for average large service.\n     If you need only some ports save memory by please patching string in decode-pcap.py \n        \"ports = None #set([80,8080]) #uncomment this if yuo don't need all traffic \"\n\n     In yandex we have used map/reduce version of this program, it were using proprietary map/ruduce implementation, \n     so it useless outside. But I waht to port in to Hadoop, if you have lot of data and test stand ask me)\n```\n\nAfter parsing there will be files http.txt and debug.txt \n\n#### Output format ####\n\n\nColumns of http.txt are the folowing:\n\n```\nserver  \nserver_port     \nclient  \nclient_port     \nrequest                 - GET/POST/HEAD\nuri \nresponse                - HTTP\nstatus                  - 200/301/500/etc\nreqid                   - Yandex Specific request id. Will be None in your case. \n\nrequest_start_time      - timestamp, first packet of request\nrequest_end_time        - timestamp, last packet of request, usually same as first, unless it is large POST request.\n\nresponse_start_time     - timestamp, first packet of response\nresponse_end_time       - timestamp, last packet of response   \n\nrequest_start_acked     - time then your server acked request first packet\nrequest_end_acked       - time then your server acked request last packet      \n\nresponse_start_acked    - time then user acked delivery of start of data\nresponse_end_acked      - time then user acked delivery of end of data\n\nrequest_retr            - number of retransmission in request     \nrequest_false_retr      - broken   \nrequest_keepalive_retr  - always 0, we track keepalive retransmission but they are not a part of request \n\n\nresponse_retr           - number of retransmission in response\nresponse_false_retr     - broken\nresponse_keepalive_retr - always 0, we track keepalive retransmission but they are not a part of response\n\n\n\nrequest_min_rtt         - different round trip time statistic by seq/ack\nrequest_median_rtt      \nrequest_max_rtt \nresponse_min_rtt        \nresponse_median_rtt     \nresponse_max_rtt        \nreq_avg_retr_time       \nresp_avg_retr_time      \n\n\nreqpackets              - number of packets in request. Usually 1.\nreqlen                  - length of payload of request\nreqreallen              - length of payload without retransmission.\n\n\nresppackets             - number of packets in response.\nresplen                 - length of payload of response\nrespreallen             - length of payload without retransmission.\n\n\nreq_user_agent          - user agent. Split by this to distinguish mobile/nonmobile traffic\nreq_host                - host from HTTP protocol\n\n\nserver_as               - broken\nserver_24mask           - broken\nclient_as               - broken \nclient_24mask           - broken\n```\n\n#### Intersing thing to calculate yourself ####\n\n``` \n   server_ack_delay = response_start_acked - request_start_time \n   #if server_ack_delay\u003e0 you have network problems (in case of server side dump)\n\n   server_latency = response_start_time - request_end_time\n   #compare to your logs, you may be surprised\n\n   request_traffic_waste = reqlen/reqreallen\n   #should be 1. If not you network is broken.\n\n   response_traffic_waste = resplen/respreallen\n   #will be more than 1. This is packet loss to users. Monitor to measure you network connectivity effectiveness\n\n   response_packets_waste = resppackets/(resplen/1450)\n   #if more than 1 - your traffic is not ideally splitted by packets. Usually on the border of jumbo frame.\n\n   response_pushthru = response_end_time - response_start_time\n   #Time of delivering to user. Usually significally more than server_latency. If it very large you probably need CDN.\n   #also could be optimized by decreasing html size, or increasing initial cwnd (don't do last one unless you really know what are you doing)\n\n   total_user_wait_time = response_end_acked - request_start_time\n   #real request start time is request_start_time-rtt, real last byte time is response_end_acked-rtt.\n   #rtt is negate itself if it stable\n\n   Also take note you could parse traffic to USER and to BACKEND and compare it. There is lot of intersting statistic here too. You just need to be able to glue request - yandex used reqid for it.\n```\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyandex%2Ftcplanz","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyandex%2Ftcplanz","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyandex%2Ftcplanz/lists"}