{"id":20115499,"url":"https://github.com/asphaltt/pkt-stucker","last_synced_at":"2026-05-09T21:37:41.048Z","repository":{"id":183981062,"uuid":"671049506","full_name":"Asphaltt/pkt-stucker","owner":"Asphaltt","description":"An experiment to reproduce the issue packet stuck in lockless pfifo_fast qdisc. [FAILED]","archived":false,"fork":false,"pushed_at":"2023-08-14T14:19:02.000Z","size":2307,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-02T19:19:40.614Z","etag":null,"topics":["ebpf","pfifo","tc-qdisc"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Asphaltt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-26T12:30:41.000Z","updated_at":"2023-12-07T08:19:58.000Z","dependencies_parsed_at":null,"dependency_job_id":"9d1017d1-48a4-40e8-bf11-f55b6abf6839","html_url":"https://github.com/Asphaltt/pkt-stucker","commit_stats":null,"previous_names":["asphaltt/pkt-stucker"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Asphaltt/pkt-stucker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Asphaltt%2Fpkt-stucker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Asphaltt%2Fpkt-stucker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Asphaltt%2Fpkt-stucker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Asphaltt%2Fpkt-stucker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Asphaltt","download_url":"https://codeload.github.com/Asphaltt/pkt-stucker/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Asphaltt%2Fpkt-stucker/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":27323899,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-28T02:00:06.623Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ebpf","pfifo","tc-qdisc"],"created_at":"2024-11-13T18:35:28.116Z","updated_at":"2025-11-28T21:01:49.836Z","avatar_url":"https://github.com/Asphaltt.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Make packets stuck in lockless pfifo_fast qdisc\n\nWhen I read the discussion about [net: sched: fix packet stuck problem for lockless qdisc](https://lore.kernel.org/lkml/1620868260-32984-2-git-send-email-linyunsheng@huawei.com/) on Twitter, an idea lingered in my mind: why not use eBPF to reproduce the issue?\n\n## Read the f\\*\\*king source code\n\nAbove all, I had to read the kernel source code to figure out how does the pfifo_fast qdisc work.\n\n```c\n__dev_queue_xmit()    // ${KERNEL}/net/core/dev.c\n|--\u003e__dev_xmit_skb() {\n        if (q-\u003eflags \u0026 TCQ_F_NOLOCK) {\n            rc = q-\u003eenqueue(skb, q, \u0026to_free) \u0026 NET_XMIT_MASK;\n            qdisc_run(q);\n            if (unlikely(to_free))\n                kfree_skb_list(to_free);\n            return rc;\n        }\n    }\n    |--\u003eqdisc_run()   // ${KERNEL}/include/net/pkt_sched.h\n        |--\u003eqdisc_run_begin() {\n        |       if (qdisc-\u003eflags \u0026 TCQ_F_NOLOCK) {\n        |           if (!spin_trylock(\u0026qdisc-\u003eseqlock))\n        |               return false;\n        |           WRITE_ONCE(qdisc-\u003eempty, false);\n        |       }\n        |   }\n        |--\u003e__qdisc_run()   // ${KERNEL}/net/sched/sch_generic.c\n        |   |--\u003eqdisc_restart()\n        |       |--\u003esch_direct_xmit()\n        |           |--\u003edev_hard_start_xmit()   // ${KERNEL}/net/core/dev.c\n        |               |--\u003exmit_one() {\n        |                       trace_net_dev_start_xmit(skb, dev);\n        |                       rc = netdev_start_xmit(skb, dev, txq, more);\n        |                       trace_net_dev_xmit(skb, rc, dev, len);\n        |                   }\n        |--\u003eqdisc_run_end() {\n                if (qdisc-\u003eflags \u0026 TCQ_F_NOLOCK)\n                    spin_unlock(\u0026qdisc-\u003eseqlock);\n            }\n```\n\nWith this code snippet, the issue may be a little easy to reproduce by making a packet stuck in `sch_direct_xmit()`.\n\n## Design the experiment\n\n![Patckets stuck in pfifo_fast qdisc](./pkt-stucker.png)\n\nThe key is to make packets stuck in `sch_direct_xmit()`.\n\nWith recognizing some tracepoints/kprobes in `sch_direct_xmit()` procedure, it’s easy to cost some CPU with eBPF in the tracepoints/kprobes.\n\nThen, intervally, two `goroutine`s do send ICMP ECHO packets to do ping.\n\nSo, as expected, one packet should be stuck in `sch_direct_xmit()`, and the other one should be enqueued and the packet handing should finish at `qdisc_run_begin()` then return early.\n\n\u003e The experiment environment:\n\u003e\n\u003e It’s a Virtualbox VM with Debian 11 (bullseye) system with 4 CPU.\n\n```shell\n# cat /etc/os-release\nPRETTY_NAME=\"Debian GNU/Linux 11 (bullseye)\"\nNAME=\"Debian GNU/Linux\"\nVERSION_ID=\"11\"\nVERSION=\"11 (bullseye)\"\nVERSION_CODENAME=bullseye\nID=debian\nHOME_URL=\"https://www.debian.org/\"\nSUPPORT_URL=\"https://www.debian.org/support\"\nBUG_REPORT_URL=\"https://bugs.debian.org/\"\n\n# uname -a\nLinux hwang 5.10.0-8-amd64 #1 SMP Debian 5.10.46-4 (2021-08-03) x86_64 GNU/Linux\n\n# llc --version\nHomebrew LLVM version 15.0.7\n  Optimized build.\n  Default target: x86_64-apple-darwin22.5.0\n  Host CPU: skylake\n\n  Registered Targets:\n    aarch64    - AArch64 (little endian)\n    aarch64_32 - AArch64 (little endian ILP32)\n    aarch64_be - AArch64 (big endian)\n    amdgcn     - AMD GCN GPUs\n    arm        - ARM\n    arm64      - ARM64 (little endian)\n    arm64_32   - ARM64 (little endian ILP32)\n    armeb      - ARM (big endian)\n    avr        - Atmel AVR Microcontroller\n    bpf        - BPF (host endian)\n    bpfeb      - BPF (big endian)\n    bpfel      - BPF (little endian)\n    hexagon    - Hexagon\n    lanai      - Lanai\n    mips       - MIPS (32-bit big endian)\n    mips64     - MIPS (64-bit big endian)\n    mips64el   - MIPS (64-bit little endian)\n    mipsel     - MIPS (32-bit little endian)\n    msp430     - MSP430 [experimental]\n    nvptx      - NVIDIA PTX 32-bit\n    nvptx64    - NVIDIA PTX 64-bit\n    ppc32      - PowerPC 32\n    ppc32le    - PowerPC 32 LE\n    ppc64      - PowerPC 64\n    ppc64le    - PowerPC 64 LE\n    r600       - AMD GPUs HD2XXX-HD6XXX\n    riscv32    - 32-bit RISC-V\n    riscv64    - 64-bit RISC-V\n    sparc      - Sparc\n    sparcel    - Sparc LE\n    sparcv9    - Sparc V9\n    systemz    - SystemZ\n    thumb      - Thumb\n    thumbeb    - Thumb (big endian)\n    ve         - VE\n    wasm32     - WebAssembly 32-bit\n    wasm64     - WebAssembly 64-bit\n    x86        - 32-bit X86: Pentium-Pro and above\n    x86-64     - 64-bit X86: EM64T and AMD64\n    xcore      - XCore\n```\n\n## Unexpected experiment case\n\nBut, the world does not work expectedly.\n\n```shell\n# ./pkt-stucker -r 192.168.1.2\n2023/07/26 22:29:42.119848 Attached tracepoint(net:net_dev_start_xmit))\n2023/07/26 22:29:42.120646 Attached tracepoint(net:net_dev_xmit))\n2023/07/26 22:29:42.122134 Listening events...\n2023/07/26 22:29:42.122615 Ticking every 10ms to send packets\n2023/07/26 22:29:42.122921 Recving packets\n2023/07/26 22:29:42.123057 Sending packets on CPU 1 in stuck1 netns\n2023/07/26 22:29:42.129358 Sending packets on CPU 3 in stuck1 netns\n2023/07/26 22:29:42.133930 Sent packet on CPU 1 with seq 100, cost 538.504µs\n256 bytes from 192.168.1.2: icmp_seq=100 time=974.026µs\n2023/07/26 22:29:42.147394 Sent packet on CPU 1 with seq 102, cost 503.232µs\n256 bytes from 192.168.1.2: icmp_seq=102 time=75.845895ms (Bingo)\n2023/07/26 22:29:42.195977 Sent packet on CPU 3 with seq 101, cost 672.386µs\n2023/07/26 22:29:42.226659 Error: found a packet cost 75.845895ms\n\n# cat /sys/kernel/debug/tracing/trace_pipe\n     pkt-stucker-332947  [001] d... 18578.382391: bpf_trace_printk: net_dev_start_xmit on CPU 1, seq: 100, ts: 18582188406462\n     pkt-stucker-332947  [001] d... 18578.382629: bpf_trace_printk: net_dev_xmit on CPU 1, seq: 100, ts: 18582188650031\n     pkt-stucker-332947  [001] d... 18578.395846: bpf_trace_printk: net_dev_start_xmit on CPU 1, seq: 102, ts: 18582201870819\n     pkt-stucker-332947  [001] d... 18578.396040: bpf_trace_printk: net_dev_xmit on CPU 1, seq: 102, ts: 18582202066973\n     pkt-stucker-332943  [003] d... 18578.444378: bpf_trace_printk: net_dev_start_xmit on CPU 3, seq: 101, ts: 18582250428892\n     pkt-stucker-332943  [003] d... 18578.444644: bpf_trace_printk: net_dev_xmit on CPU 3, seq: 101, ts: 18582250695454\n```\n\n## Run the demo\n\nWhen the 4-CPUs VM prepares, run the demo:\n\n```shell\n# apt install -y git clang-15 llvm-15\n# git clone https://github.com/Asphaltt/pkt-stucker.git\n# cd pkt-stucker\n# go generate\n# go build\n# bash setup-env.sh\n# echo \"bash clear-env.sh finally\"\n# ./pkt-stucker -r 192.168.1.2\n#\n# echo In another terminal\n# cat /sys/kernel/debug/tracing/trace_pipe\n```\n\n## In conclusion\n\nWow, congrad, the issue is reproduced.\n\nFake it. This is a way to reproduce the issue.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasphaltt%2Fpkt-stucker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fasphaltt%2Fpkt-stucker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasphaltt%2Fpkt-stucker/lists"}