{"id":16724544,"url":"https://github.com/yunwei37/nginx-lua-ebpf-toolkit","last_synced_at":"2025-04-09T18:21:50.406Z","repository":{"id":43905687,"uuid":"497654748","full_name":"yunwei37/nginx-lua-ebpf-toolkit","owner":"yunwei37","description":"profile and tracking tools for lua and nginx using eBPF","archived":false,"fork":false,"pushed_at":"2024-11-07T06:23:07.000Z","size":2296,"stargazers_count":64,"open_issues_count":2,"forks_count":14,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-02T13:54:51.294Z","etag":null,"topics":["ebpf","lua","nginx","openresty","uprobes"],"latest_commit_sha":null,"homepage":"https://github.com/apache/apisix-profiler","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yunwei37.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-05-29T17:03:22.000Z","updated_at":"2025-03-23T04:11:25.000Z","dependencies_parsed_at":"2024-10-27T11:51:35.926Z","dependency_job_id":"14ffcb2e-7316-4deb-a288-a57abb43d99a","html_url":"https://github.com/yunwei37/nginx-lua-ebpf-toolkit","commit_stats":{"total_commits":35,"total_committers":2,"mean_commits":17.5,"dds":0.05714285714285716,"last_synced_commit":"b0477fe4c2ea768fe385115967dece50824888a8"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yunwei37%2Fnginx-lua-ebpf-toolkit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yunwei37%2Fnginx-lua-ebpf-toolkit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yunwei37%2Fnginx-lua-ebpf-toolkit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yunwei37%2Fnginx-lua-ebpf-toolkit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yunwei37","download_url":"https://codeload.github.com/yunwei37/nginx-lua-ebpf-toolkit/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248085618,"owners_count":21045189,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ebpf","lua","nginx","openresty","uprobes"],"created_at":"2024-10-12T22:45:40.809Z","updated_at":"2025-04-09T18:21:50.386Z","avatar_url":"https://github.com/yunwei37.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ebpf-based lua profile tools\n\nUse ebpf to generate lua flamegraphs:\n\n- trace lua stack in kernel\n- zero code changes or restarts needed\n- support `luajit 32/luajit 64`\n- working well on new kernel(\u003e=5.13)\n- faster speed\n- run only a small binary without any dependencies\n\n\u003e Note: \n\u003e - for the ebpf verifier instructions limit in kernel, the stack-trace deepth is limited to top 15 in lua. If you need to trace deeper, you need to use systemtap instead.\n\u003e - this project is not finished yet, and some errors may occurred.\n\nThe docker image can be found in:\n\n```sh\ndocker pull ghcr.io/yunwei37/nginx-lua-profile:latest\n```\n\n## probe lua stack in nginx\n\nsee: bpftools/profile_nginx_lua/profile.bpf.c\n\nfirst, we use `uprobe` in ebpf to attach to `libluajit.so` get the lua_State:\n\n```c\nstatic int probe_entry_lua(struct pt_regs *ctx)\n{\n\tif (!PT_REGS_PARM1(ctx))\n\t\treturn 0;\n\n\t__u64 pid_tgid = bpf_get_current_pid_tgid();\n\t__u32 pid = pid_tgid \u003e\u003e 32;\n\t__u32 tid = (__u32)pid_tgid;\n\tstruct lua_stack_event event = {};\n\n\tif (targ_pid != -1 \u0026\u0026 targ_pid != pid)\n\t\treturn 0;\n\t// event.time = bpf_ktime_get_ns();\n\tevent.pid = pid;\n\t// bpf_get_current_comm(\u0026event.comm, sizeof(event.comm));\n\tevent.L = (void *)PT_REGS_PARM1(ctx);\n\t// bpf_printk(\"lua_state %p\\n\", event.L);\n\tbpf_map_update_elem(\u0026lua_events, \u0026tid, \u0026event, BPF_ANY);\n\treturn 0;\n}\n```\n\nto get stack frame of lua, it uses a loop to backtrace the lua vm stack and find all information of functions:\n\nsee the `fix_lua_stack` function:\n```c\n\t....\n\tcTValue *frame, *nextframe, *bot = tvref(BPF_PROBE_READ_USER(L, stack)) + LJ_FR2;\n\tint i = 0;\n\tframe = nextframe = BPF_PROBE_READ_USER(L, base) - 1;\n\t/* Traverse frames backwards. */\n\t// for the ebpf verifier insns (limit 1000000), we need to limit the max loop times to 15\n\tfor (; i \u003c 15 \u0026\u0026 frame \u003e bot; i++)\n\t{\n\t\tif (frame_gc(frame) == obj2gco(L))\n\t\t{\n\t\t\tlevel++; /* Skip dummy frames. See lj_err_optype_call(). */\n\t\t}\n\t\tif (level-- == 0)\n\t\t{\n\t\t\tlevel++;\n\t\t\t// *size = (nextframe - frame);\n\t\t\t/* Level found. */\n\t\t\tif (lua_get_funcdata(ctx, frame, eventp, count) != 0)\n\t\t\t{\n\t\t\t\tcontinue;\n\t\t\t}\n\t\t\tcount++;\n\t\t}\n\t\tnextframe = frame;\n\t\tif (frame_islua(frame))\n\t\t{\n\t\t\tframe = frame_prevl(frame);\n\t\t}\n\t\telse\n\t\t{\n\t\t\tif (frame_isvarg(frame))\n\t\t\t\tlevel++; /* Skip vararg pseudo-frame. */\n\t\t\tframe = frame_prevd(frame);\n\t\t}\n\t}\n\t....\n\n```\n\nafter that, it gets the function data send the function data to user space:\n\n```c\nstatic inline int lua_get_funcdata(struct bpf_perf_event_data *ctx, cTValue *frame, struct lua_stack_event *eventp, int level)\n{\n\tif (!frame)\n\t\treturn -1;\n\tGCfunc *fn = frame_func(frame);\n\tif (!fn)\n\t\treturn -1;\n\tif (isluafunc(fn))\n\t{\n\t\teventp-\u003etype = FUNC_TYPE_LUA;\n\t\tGCproto *pt = funcproto(fn);\n\t\tif (!pt)\n\t\t\treturn -1;\n\t\teventp-\u003effid = BPF_PROBE_READ_USER(pt, firstline);\n\t\tGCstr *name = proto_chunkname(pt); /* GCstr *name */\n\t\tconst char *src = strdata(name);\n\t\tif (!src)\n\t\t\treturn -1;\n\t\tbpf_probe_read_user_str(eventp-\u003ename, sizeof(eventp-\u003ename), src);\n\t\tbpf_printk(\"level= %d, fn_name=%s\\n\", level, eventp-\u003ename);\n\t}\n\telse if (iscfunc(fn))\n\t{\n\t\teventp-\u003etype = FUNC_TYPE_C;\n\t\teventp-\u003efuncp = BPF_PROBE_READ_USER(fn, c.f);\n\t}\n\telse if (isffunc(fn))\n\t{\n\t\teventp-\u003etype = FUNC_TYPE_F;\n\t\teventp-\u003effid = BPF_PROBE_READ_USER(fn, c.ffid);\n\t}\n\teventp-\u003elevel = level;\n\tbpf_perf_event_output(ctx, \u0026lua_event_output, BPF_F_CURRENT_CPU, eventp, sizeof(*eventp));\n\treturn 0;\n}\n```\n\nin user space, it will use the `user_stack_id` to mix the lua stack with the original user and kernel stack:\n\nsee `bpftools/profile_nginx_lua/profile.c: print_fold_user_stack_with_lua`\n```\n\t\t\t\t....\n\t\t\t\tconst struct lua_stack_event* eventp = \u0026(lua_bt-\u003estack[count]);\n\t\t\t\tif (eventp-\u003etype == FUNC_TYPE_LUA)\n\t\t\t\t{\n\t\t\t\t\tif (eventp-\u003effid) {\n\t\t\t\t\t\tprintf(\";L:%s:%d\", eventp-\u003ename, eventp-\u003effid);\n\t\t\t\t\t} else {\n\t\t\t\t\t\tprintf(\";L:%s\", eventp-\u003ename);\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\telse if (eventp-\u003etype == FUNC_TYPE_C)\n\t\t\t\t{\n\t\t\t\t\tsym = syms__map_addr(syms, (unsigned long)eventp-\u003efuncp);\n\t\t\t\t\tif (sym)\n\t\t\t\t\t{\n\t\t\t\t\t\tprintf(\";C:%s\", sym ? sym-\u003ename : \"[unknown]\");\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\telse if (eventp-\u003etype == FUNC_TYPE_F)\n\t\t\t\t{\n\t\t\t\t\tprintf(\";builtin#%d\", eventp-\u003effid);\n\t\t\t\t}\n\t\t\t\t....\n```\n\nIf the lua stack output `user_stack_id` matches the original `user_stack_id`, this means the stack is a lua stack. Then, we replace the `[unknown]` function whose uip insides the luajit vm function range with our lua stack. This may not be totally correct, but it works for now. After printing the stack, we can use \n\n## results flamegraph\n\nlua:\n\n![flamegraph](bpftools/profile_nginx_lua/results/user-lua.svg)\n\nlua and c:\n\n![flamegraph](bpftools/profile_nginx_lua/results/user-lua-c.svg)\n\n### reference\n\nfor reference, I looked into the debug functions of lua vm:\n\n```c\nLJ_FUNC void lj_debug_dumpstack(lua_State *L, SBuf *sb, const char *fmt,\n\t\t\t\tint depth);\n```\n\nin `lj_debug.h` and `lj_debug.c` from luajit source code: `openresty-1.21.4.1/build/LuaJIT-2.1-20220411/src/lj_debug.h`  \n\nfor lua data structure definition, see: `bpftools/profile_nginx_lua/lua_state.h`, it's copied from `luajit` headers.\n\nwe can determine the `luajit gc32/64` from:\n\nbpftools/profile_nginx_lua/lua_state.h:9\n```c\n#define LJ_TARGET_GC64 1\n```\n\nthe openresty used and tested is from `https://openresty.org/en/benchmark.html`, and the `APISIX` used is https://github.com/apache/apisix\n\nand:\n\n- https://github.com/api7/stapxx/blob/master/tapset/luajit_gc64.sxx\n- https://github.com/openresty/openresty-systemtap-toolkit/blob/master/ngx-sample-lua-bt\n\nThe ebpf program is from: https://github.com/iovisor/bcc/pull/3782\n\n### to run lua profile:\n\ntested with `luajit-5.1.so` gc64:\n\nfor example, use apisix profile scripts in CI to start `APISIX`(ci/performance_test.sh):\n\n```bash\n# get nginx pid\npgrep -P $(cat logs/nginx.pid) -n -f worker\n# sample only user stack and lua stack, use fold output, trace pid 36685 for nginx\ncd bpftools/profile_nginx_lua\nmake\nsudo ./profile -f -F 499 -U -p [pid] --lua-user-stacks-only \u003e a.bt\n# get flame graph\ncat a.bt | ../../tools/FlameGraph/flamegraph.pl \u003e a.svg\n```\n\n## test when running benchmark\n\nbasic benchmark：\n\n- https://openresty.org/en/getting-started.html\n- https://openresty.org/en/benchmark.html\n\n```\ncd ~/work\nPATH=/usr/local/openresty/nginx/sbin:$PATH\nexport PATH\nnginx -p `pwd`/ -c conf/nginx.conf\n```\n\n```\ncurl http://localhost:8080/\n```\n\nor using APISIX performance test: \n\n```\ngit clone https://github.com/apache/apisix\ncd apisix\nsudo apt-get install -y libpcre3 libpcre3-dev\nsudo apt-get install -y openssl libssl-dev unzip zlib*\nsudo ./ci/performance_test.sh install_dependencies\nsudo ./ci/performance_test.sh install_wrk2\nsudo ./ci/performance_test.sh install_stap_tools\n./ci/performance_test.sh run_performance_test\n```\n\n## test with containers or missing debug info\n\nThis tool can get the APISIX process within a container, and get some stack trace ip. \nHowever, If the debug info is not shipped with the docker, the tool cannot use the uprobe\nto trace the lua stack, and the c stack also cannot get any valid output. Solutions may be:\n\n- get some debug info for the APISIX program in docker, then it can work with both c and lua tracing.\n  If the program cannot find the debug info correctly, we may need to specify it mannually.\n- Compile luajit with some user-space tracepoints (see: https://lwn.net/Articles/753601/), then we \n  can use this for uprobe to trace lua stacks, but c stacks still cannot work.\n\n## for APISIX OSPP\n\n### 项目产出要求：\n\n使用eBPF捕获和解析 `Apache APISIX` 中的 lua 调用堆栈信息，对其进行汇总并生成cpu火焰图：\n- 利用eBPF同时捕获和解析C和Lua混合调用堆栈信息，对其进行总结，生成cpu火焰图。\n- 支持获取在Docker中运行的 `Apache APISIX` 进程\n- 支持获取 Apache APISIX Openresty luajit 32/luajit 64 模式\n\nTODO:\n\n- [ ] test in docker\n- [ ] test for more version of luajit32/64\n- [ ] add more functionalities\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyunwei37%2Fnginx-lua-ebpf-toolkit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyunwei37%2Fnginx-lua-ebpf-toolkit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyunwei37%2Fnginx-lua-ebpf-toolkit/lists"}