{"id":16693453,"url":"https://github.com/paulross/dtrace-py","last_synced_at":"2025-03-21T19:30:50.126Z","repository":{"id":84163335,"uuid":"100302268","full_name":"paulross/dtrace-py","owner":"paulross","description":"Using a Dtrace with Python","archived":false,"fork":false,"pushed_at":"2018-09-17T11:33:45.000Z","size":1744,"stargazers_count":57,"open_issues_count":0,"forks_count":3,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-18T04:12:34.563Z","etag":null,"topics":["debugging","dtrace","lightning-talk","python","tracing"],"latest_commit_sha":null,"homepage":null,"language":"DTrace","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/paulross.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-08-14T19:28:00.000Z","updated_at":"2023-04-03T15:09:05.000Z","dependencies_parsed_at":"2023-03-01T08:00:36.334Z","dependency_job_id":null,"html_url":"https://github.com/paulross/dtrace-py","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulross%2Fdtrace-py","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulross%2Fdtrace-py/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulross%2Fdtrace-py/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/paulross%2Fdtrace-py/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/paulross","download_url":"https://codeload.github.com/paulross/dtrace-py/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244855306,"owners_count":20521617,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["debugging","dtrace","lightning-talk","python","tracing"],"created_at":"2024-10-12T16:30:37.948Z","updated_at":"2025-03-21T19:30:49.430Z","avatar_url":"https://github.com/paulross.png","language":"DTrace","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dtrace and Python\n\nThis is a demonstration of using Dtrace with Python 3.6/3.7 suitable for a five minute lightning talk. You need an Apple Mac or another OS with Dtrace support.\n\nThe demonstration shows how dtrace can attach and profile running Python processes giving you live profiling and debugging information.\n\nThe fiction surrounding this demonstration is of a hot new startup that creates personalised cat videos and it wants to demonstrate its SRE skills by tracing a server in production, live!\n\n# Preparation\n\nCreate a dtrace version of Python and a virtual environment in `~/venvs` directory (or wherever you prefer):\n\n```\ncd ~/tmp\ncurl -o Python-3.7.0.tgz https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tgz\ntar -xzf Python-3.7.0.tgz\ncd Python-3.7.0\n./configure --with-dtrace\nmake\npython.exe -m venv ~/venvs/dtrace\n```\n\nIf you want to show off Py-Spy then install it:\n\n```\n(dtrace36)\npaulross@Pauls-MBP-1  ~/dtrace-py (master)\n$ pip install py-spy\nCollecting py-spy\n  Using cached https://files.pythonhosted.org/packages/67/77/57fbee60cb6870894a9436a80ba5e19bd46cc7d040615e5ab07863d1726a/py_spy-0.1.5-py2.py3-none-macosx_10_7_x86_64.whl\nInstalling collected packages: py-spy\nSuccessfully installed py-spy-0.1.5\n```\n\n\n## Setup\n\nCreate three shells, `cd` to the directory this README.md is in.\n\nIn the first ('Python') shell, make it around 60 columns wide, activate dtrace Python3.6 and launch it, this will be used to run the python demo continuously:\n\n```\n. ~/venvs/dtrace/bin/activate\npython3\n```\n\nIn the second ('Dtrace') shell put it side by side with the 'Python' shell, if you can make it 100 columns wide that would be great.\n\nIn a third ('Presentation') shell make it 85 columns wide with the text as large as possible for the presentation.\n\n## Checks\n\nCheck that the changes of `d_demo/a_py_flowinfo.d` from a previous demo in function-entry and function-return are absent, the line `/copyinstr(arg1) == \"new_cat_video\"/` should not be in the functions `python*:::function-entry` and `python*:::function-return`:\n\nCheck that the version of Python you are running has probes, I'm assuming PID 24601 here:\n\n```\n$ sudo dtrace -l -P python24601\nPassword:\n   ID   PROVIDER            MODULE                          FUNCTION NAME\n 4875 python24601          python3          _PyEval_EvalFrameDefault function-entry\n 4876 python24601          python3          _PyEval_EvalFrameDefault function-return\n 4877 python24601          python3                           collect gc-done\n 4878 python24601          python3                           collect gc-start\n 4879 python24601          python3          _PyEval_EvalFrameDefault line\n```\n\n# The Lightning Talk\n\nIn the presentation shell run:\n\n```\n$ python3 presentation.py\n```\n\nThe following keys work:\n\n* `\u003ccr\u003e`  - Next slide.\n* `b\u003ccr\u003e` - Previous slide.\n* `q\u003ccr\u003e` - Quit\n* `r\u003ccr\u003e` - Refresh display (useful after resizing the command line).\n\nWhen you reach the \"Live demo!\" slide bring the other two shells to the foreground.\n\nIn the dtrace shell bring up the `demo.py` code:\n\n![](images/demo.py.png)\n\n\"This is a demo of a hot new startup that will deliver to you every day a personalised cat video. It examines your social media profile and searches the web for just the right cat video for you. Of course this is just a MVP for angel investors.\n\nIn the code above, `go()` represents the server continuously running. It simulates serving 8 requests for cat videos by calling `find()` with a random number to represent the search depth. `find()` then calls `new_cat_video()` that randomly pauses to represent the hard work of personalising a cat video. Its really just a simulation of our server at work.\"\n\nIn the Python shell launch Python, then start the server with:\n\n```\n\u003e\u003e\u003e import demo\n\u003e\u003e\u003e demo.go()\n```\n\nYou should see something like this, the PID is important for the next step:\n\n![](images/DemoRunning.png)\n\nEach `*` represents part of the search and each `+` represents a cat video being personalised.\n\n\"Now one of our SRE's want to trace what is going on in the production server.\"\n\n## Tracing All Python Function Calls\n\nIn the dtrace shell:\n\n```\n$ sudo dtrace -s d_demo/a_py_flowinfo.d -p 24601\n```\n\nYou should see something like this [`Ctrl-C` to stop tracing]:\n\n![](images/A_TracingFunctions.png)\n\nOn the right we have the following columns for each function call or return:\n\n1. Core number.\n2. PID.\n3. Delta time in microseconds from previous line.\n4. Source file and line number.\n5. Type of the probe, in all these cases it is func but garbage collection start/stop is also possible.\n6. Function, indented by stack depth.\n\nThis even traces the standard library of course.\n\n\"I don't want the whole call stack, its too much, just give me the `new_cat_video()` function call.\"\n\n## Tracing a Specific Function\n\nYou can edit `d_demo/a_py_flowinfo.d` to show only the function `new_cat_video()` by copying line 53 which is the predicate: `/copyinstr(arg1) == \"new_cat_video\"/`\n\n```\n$ vi d_demo/a_py_flowinfo.d\n```\n\nAnd inserting this predicate as lines 75 and 86:\n\n![](images/vi_edit.png)\n\nRunning the same command:\n\n```\n$ sudo dtrace -s d_demo/a_py_flowinfo.d -p 24601\n```\nYou should see something like this:\n\n![](images/A_TracingFunctions_2.png)\n\n## Function Counting\n\nThis aggregates the the function calls and the execution time. Use the command:\n\n```\n$ sudo dtrace -s d_demo/b_py_calltime.d -p 24601\n```\n\nWait for a few seconds then hit `Ctrl-C` and you will see:\n\n![](images/B_FunctionCount.png)\n\nThis gives the count of function calls and the inclusive and exclusive time spent in them in microseconds.\n\n\"But all those numbers, can't we have pictures?\"\n\n## Function Execution Time Histogram\n\nWith the command:\n\n```\n$ sudo dtrace -s d_demo/c_py_calldist.d -p 24601\n```\n\nYou should see an ASCII histogram, note `demo.py, func, new_cat_video`, the second table from the bottom:\n\n![](images/C_TimeHistogram.png)\n\nThat is the demo over, now back to the presentation shell and finish that off.\n\nWith a little practice this can all be done in less than 5 minutes.\n\n# Py-Spy\n\nIf you are quick you might be able to demonstrate Py-Spy.\n\n[Py-Spy](https://github.com/benfred/py-spy) is a sampling profiler for Python programs.\nIt lets you visualize what your Python program is spending time on without restarting the program or modifying the code in any way.\nPy-Spy is extremely low overhead: it is written in Rust for speed and doesn't run in the same process as the profiled Python program, nor does it interrupt the running program in any way.\nThis means Py-Spy is safe to use against production Python code.\n\nPy-Spy works on Linux, OSX and Windows, and supports profiling all recent versions of the CPython interpreter (versions 2.3-2.7 and 3.3-3.6).\n\nHere is Py-Spy sampling our which is running this time as PID 6871:\n```\n$ sudo py-spy -p 6871\n```\n\n![Py-Spy top](images/PySpyTop.png)\n\nAnd the interactive SVG flame graph obtained by:\n\n```\n$ sudo py-spy -p 6871 -f images/PySpyFlame.svg\n```\n\n![Py-Spy Flame graph](images/PySpyFlame.svg)\n\n\n# Further Resources\n\n* Dtrace on [Wikipedia](http://en.wikipedia.org/wiki/DTrace)\n* The Dtrace [website](http://dtrace.org/blogs/).\n* Brendan Gregg on [Dtrace](http://www.brendangregg.com/dtrace.html) and the [Dtrace toolkit](http://www.brendangregg.com/dtracetoolkit.html).\n* Brendan on [eBPF](http://www.brendangregg.com/ebpf.html), the Linux equivalent ot Dtrace.\n* Py-Spy [home page](https://github.com/benfred/py-spy)\n\nIn this repo you will find a other Dtrace `*.d` files for Python in the `toolkit/` directory.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaulross%2Fdtrace-py","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpaulross%2Fdtrace-py","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpaulross%2Fdtrace-py/lists"}