{"id":13655202,"url":"https://github.com/felixge/fgtrace","last_synced_at":"2025-05-15T10:06:57.154Z","repository":{"id":59645677,"uuid":"537923574","full_name":"felixge/fgtrace","owner":"felixge","description":"fgtrace is an experimental profiler/tracer that is capturing wallclock timelines for each goroutine. It's very similar to the Chrome profiler.","archived":false,"fork":false,"pushed_at":"2022-11-27T12:24:09.000Z","size":2167,"stargazers_count":910,"open_issues_count":0,"forks_count":14,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-05-14T14:20:02.403Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/felixge.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-09-17T20:31:13.000Z","updated_at":"2025-05-12T21:36:32.000Z","dependencies_parsed_at":"2023-01-22T12:30:59.696Z","dependency_job_id":null,"html_url":"https://github.com/felixge/fgtrace","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/felixge%2Ffgtrace","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/felixge%2Ffgtrace/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/felixge%2Ffgtrace/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/felixge%2Ffgtrace/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/felixge","download_url":"https://codeload.github.com/felixge/fgtrace/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254319720,"owners_count":22051073,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T03:00:59.506Z","updated_at":"2025-05-15T10:06:52.136Z","avatar_url":"https://github.com/felixge.png","language":"Go","readme":"# fgtrace - The Full Go Tracer\n\n[![go-recipes](https://raw.githubusercontent.com/nikolaydubina/go-recipes/main/badge.svg?raw=true)](https://github.com/nikolaydubina/go-recipes)\n[![ci test status](https://img.shields.io/github/workflow/status/felixge/fgtrace/Go?label=tests)](https://github.com/felixge/fgtrace/actions/workflows/go.yml?query=branch%3Amain)\n[![documentation](http://img.shields.io/badge/godoc-reference-blue.svg)](https://pkg.go.dev/github.com/felixge/fgtrace)\n\nfgtrace is an experimental profiler/tracer that is capturing wallclock timelines for each goroutine. It's very similar to the Chrome profiler.\n\n⚠️ fgtrace may cause noticeable stop-the-world pauses in your applications. It is intended for dev and testing environments for now.\n\n\u003cimg src=\"./assets/fgtrace-example.png\"/\u003e\n\n## Quick Start\n\nTo capture an fgtrace of your program, simply add the one-liner shown below. This will cause the creation of a `fgtrace.json` file in the current working directory that you can view by opening it in the [Perfetto UI](https://ui.perfetto.dev/).\n\n```go\npackage main\n\nimport \"github.com/felixge/fgtrace\"\n\nfunc main() {\n\tdefer fgtrace.Config{Dst: fgtrace.File(\"fgtrace.json\")}.Trace().Stop()\n\n\t// \u003ccode to trace\u003e\n}\n```\n\nAlternatively you can configure fgtrace as a `http.Handler` and request traces on-demand by hitting `http://localhost:1234/debug/fgtrace?seconds=30\u0026hz=100`.\n\n```go\npackage main\n\nimport (\n\t\"net/http\"\n\t\"github.com/felixge/fgtrace\"\n)\n\nfunc main() {\n\thttp.DefaultServeMux.Handle(\"/debug/fgtrace\", fgtrace.Config{})\n\thttp.ListenAndServe(\":1234\", nil)\n}\n```\n\nFor more advanced use cases, have a look at the [API Documentation](https://pkg.go.dev/github.com/felixge/fgtrace#Config).\n\n## Comparison with Similar Tools\n\nBelow is a [simple program](./testdata/readme/) that spends its time sleeping, requesting a website, capturing the response body and then hashing it a few times.\n\n```go\nfor i := 0; i \u003c 10; i++ {\n\ttime.Sleep(10 * time.Millisecond)\n}\n\nres, err := http.Get(\"https://github.com/\")\nif err != nil {\n\tpanic(err)\n}\ndefer res.Body.Close()\n\nvar buf bytes.Buffer\nif _, err := io.Copy(\u0026buf, res.Body); err != nil {\n\tpanic(err)\n}\n\nfor i := 0; i \u003c 1000; i++ {\n\tsha1.Sum(buf.Bytes())\n}\n```\n\nNow let's have a look at how fgtrace and other tools allow you to understand the performance of such a program.\n\n### fgtrace\n\nLooking at our main goroutine (G1), we can easily recognize the operations of the program, their order, and how long they are taking (~100ms `time.Sleep`, ~65ms `http.Get`, ~30ms `io.Copy`ing the response and ~300ms calling `sha1.Sum` to hash it).\n\nHowever, it's important to note that this data is captured by sampling goroutine stack traces rather than actual tracing. Therefore fgtrace does not know that there were ten `time.Sleep()` function calls lasting `10ms` each. Instead it just merges its samples into one big `time.Sleep()` call that appears to take `100ms`.\n\nAnother detail are the virtual goroutine state indicators on top, e.g. `sleep`, `select`, `sync.Cond.Wait` and `running/runnable`. These are not part of the real stack traces and meant to help understanding On-CPU activity (`running/runnable`) vs Off-CPU states. You can disable them via configuration.\n\n\u003cimg src=\"./assets/fgtrace-example.png\"/\u003e\n\nTo break down the latency of our main goroutine, we can also look at other goroutines used by the program. E.g. below is a closer look on how the `http.Get` operation is broken down into resolving the IP address, connecting to it, and performing a TLS handshake.\n\n\u003cimg src=\"./assets/fgtrace-example2.png\"/\u003e\n\nSo as you can see, fgtrace offers an intuitive, yet powerful way to understand the operation of Go programs. However, since it always captures the activity of all goroutines and has no information about how they communicate with each other, it may create overwhelming amounts of data in some cases.\n\n### fgprof\n\nYou can think of [fgprof](https://github.com/felixge/fgprof) as a more simplified version of fgtrace. Instead of capturing a timeline for each goroutine, it aggregates the same data into a single profile as shown in the flame graph below.\n\n\u003cimg src=\"./assets/fgprof-example.png\"/\u003e\n\nThis means that the x-axis represents duration rather than time, so function calls are ordered alphabetically rather than chronologically. E.g. notice how `time.Sleep` is shown after `sha1.Sum` in the graph above even so it's the first operation completed by our program.\n\nAdditionally the data of all goroutines ends up in the same graph which can be difficult to read without having a good understanding of the underlaying code and number of goroutines that are involved.\n\nDespite these disadvantages, fgprof may still be useful in certain situations where the detail provided by the timeline may be overwhelming and a simpler view of the average program behavior is desirable. Additionally fgprof under Go 1.19 has less [negative impact](https://go-review.googlesource.com/c/go/+/387415) on the performance of the profiled program than fgtrace.\n\n### runtime/trace\n\nThe `runtime/trace` package is a true execution tracer that is capable of capturing even more detailed information than fgtrace. However, it's mostly designed to understand the decisions made by the Go scheduler. So the default timeline is focused on how goroutines are scheduled onto the CPU (processors). This means only the `sha1.Sum` operation stands out in green, and full stack traces can only be seen by clicking on the individual scheduler activities.\n\n\u003cimg src=\"./assets/runtime-example.png\"/\u003e\n\nThe goroutine analysis view offers a more useful breakdown. Here we can see that our goroutine is spending `271ms` in `Execution` on CPU, but it's not clear from this view alone that this is the `sha1.Sum` operation. Our networking activity (`http.Get` and `io.Copy`) gets grouped into `Sync block` rather than `Network wait` because the networking is done through channels via other goroutines. And our `time.Sleep` activity is shown as a grey component of the bar diagram, but not explicitly listed in the table. So while a lot of information is available here, it's difficult to interpret for casual users.\n\n\u003cimg src=\"./assets/runtime-example2.png\"/\u003e\n\nLast but not least it's possible to click on the goroutine id in the view above in order to see a timeline for the individual goroutine, as well as the other goroutines it is communicating with. However, the view is also CPU-centric, so remains difficult to understand the sleep and networking operations of our program.\n\n\u003cimg src=\"./assets/runtime-example3.png\"/\u003e\n\nThat being said, some of the limitations of `runtime/trace` could probably be resolved with changes to the UI (see [gotraceui](https://github.com/dominikh/gotraceui)) or converting the traces into a format that [Perfetto UI](https://ui.perfetto.dev/) can understand which might be a fun project for another time.\n\n## How it Works\n\nThe current implementation of fgtrace is incredibly hacky. It calls [`runtime.Stack()`](https://pkg.go.dev/runtime#Stack) on a regular frequency (default 100 Hz) to capture textual stack traces of all goroutines and parses them using the [gostackparse](https://github.com/DataDog/gostackparse) package. Each call to `runtime.Stack()` is a blocking stop-the-world operation, so it scales very poorly to programs using ten thousand or more goroutines.\n\nAfter the data is captured, it is converted into the [Trace Event Format](https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/preview) which is one of the data formats understood by [Perfetto UI](https://ui.perfetto.dev/).\n\n## The Future\n\nfgtrace is mostly a [\"Do Things that Don't Scale\"](http://paulgraham.com/ds.html) kind of project. If enough people like it, it will motivate me and perhaps others to invest into putting it on a solid technical foundation.\n\nThe Go team has previously [declined](https://github.com/golang/go/issues/41324#issuecomment-703796820) the idea of adding wallclock profiling capabilities similar to fgprof (which is similar to fgtrace) to the Go project and is more likely to invest in `runtime/trace` going forward.\n\nThat being said, I still think fgtrace can help by:\n\n1. Showing the usefulness of stack-trace/wallclock focused timeline views in addition to the CPU-centric views used by `runtime/trace`.\n2. Starting a conversation (link to GH issue will follow ...) to offer more powerful goroutine profiling APIs to allow user-space tooling like this to thrive without having to hack around the [existing APIs](https://github.com/DataDog/go-profiler-notes/blob/main/goroutine.md#feature-matrix) while reducing their overhead.\n\n\n\n## License\n\nfgtrace is licensed under the MIT License.\n","funding_links":[],"categories":["Programming Languages"],"sub_categories":["Golang"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffelixge%2Ffgtrace","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffelixge%2Ffgtrace","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffelixge%2Ffgtrace/lists"}