{"id":13582379,"url":"https://github.com/cloudflare/ebpf_exporter","last_synced_at":"2025-04-23T20:58:34.431Z","repository":{"id":41055431,"uuid":"130777949","full_name":"cloudflare/ebpf_exporter","owner":"cloudflare","description":"Prometheus exporter for custom eBPF metrics","archived":false,"fork":false,"pushed_at":"2025-04-08T04:13:36.000Z","size":13027,"stargazers_count":2312,"open_issues_count":12,"forks_count":251,"subscribers_count":51,"default_branch":"master","last_synced_at":"2025-04-23T20:58:25.312Z","etag":null,"topics":["bpf","ebpf","libbpf","linux-kernel","performance","prometheus","prometheus-exporter","tracing"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cloudflare.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-04-24T01:24:39.000Z","updated_at":"2025-04-23T20:12:19.000Z","dependencies_parsed_at":"2023-02-17T09:15:37.534Z","dependency_job_id":"c9e355de-050a-4da6-bfe0-9d43b8c8e0e3","html_url":"https://github.com/cloudflare/ebpf_exporter","commit_stats":{"total_commits":461,"total_committers":42,"mean_commits":"10.976190476190476","dds":"0.42082429501084595","last_synced_commit":"2de457ebf6e66cea519b256f246546055a480e92"},"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudflare%2Febpf_exporter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudflare%2Febpf_exporter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudflare%2Febpf_exporter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudflare%2Febpf_exporter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cloudflare","download_url":"https://codeload.github.com/cloudflare/ebpf_exporter/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250514767,"owners_count":21443208,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bpf","ebpf","libbpf","linux-kernel","performance","prometheus","prometheus-exporter","tracing"],"created_at":"2024-08-01T15:02:39.577Z","updated_at":"2025-04-23T20:58:34.407Z","avatar_url":"https://github.com/cloudflare.png","language":"Go","funding_links":[],"categories":["Go","others","Linux","Networking \u0026 Performance","Mobile"],"sub_categories":["Performance","Traffic Analysis \u0026 Filtering","Linux/ *Nix"],"readme":"# ebpf_exporter\n\nPrometheus exporter for custom eBPF metrics and OpenTelemetry traces.\n\n* Metrics:\n\n![metrics](./examples/biolatency.png)\n\n* [Traces](./tracing):\n\n![tracing](./examples/exec-trace.png)\n\nMotivation of this exporter is to allow you to write eBPF code and export\nmetrics that are not otherwise accessible from the Linux kernel.\n\n[ebpf.io](https://ebpf.io/what-is-ebpf/) describes eBPF:\n\n\u003e eBPF is a revolutionary technology with origins in the Linux kernel that can\n\u003e run sandboxed programs in a privileged context such as the operating system\n\u003e kernel. It is used to safely and efficiently extend the capabilities of the\n\u003e kernel without requiring to change kernel source code or load kernel modules.\n\nAn easy way of thinking about this exporter is bcc tools as prometheus metrics:\n\n* https://iovisor.github.io/bcc\n\nWe use libbpf rather than legacy bcc driven code, so it's more like libbpf-tools:\n\n* https://github.com/iovisor/bcc/tree/master/libbpf-tools\n\nProducing [OpenTelemetry](https://opentelemetry.io/) compatible traces is also\nsupported, see [Tracing docs](./tracing/) for more information on that.\n\n## Reading material\n\n* https://www.brendangregg.com/ebpf.html\n* https://nakryiko.com/posts/bpf-core-reference-guide/\n* https://nakryiko.com/posts/bpf-portability-and-co-re/\n* https://nakryiko.com/posts/bcc-to-libbpf-howto-guide/\n* https://libbpf.readthedocs.io/en/latest/program_types.html\n\n## Building and running\n\n### Actual building\n\nTo build a binary, clone the repo and run:\n\n```\nmake build\n```\n\nThe default `build` target makes a static binary, but you could also\nuse the `build-dynamic` target if you'd like a dynamically linked binary.\nIn either case `libbpf` is built from source, but you could override this\nbehavior with `BUILD_LIBBPF=0`, if you want to use your system `libbpf`.\n\nIf you're having trouble building on the host, you can try building in Docker:\n\n```\ndocker build --tag ebpf_exporter --target ebpf_exporter .\ndocker cp $(docker create ebpf_exporter):/ebpf_exporter ./\n```\n\nTo build examples (see [building examples section](#building-examples)):\n\n```\nmake -C examples clean build\n```\n\nTo run with [`biolatency`](examples/biolatency.yaml) config:\n\n```\nsudo ./ebpf_exporter --config.dir=examples --config.names=biolatency\n```\n\nIf you pass `--debug`, you can see raw maps at `/maps` endpoint\nand see debug output from `libbpf` itself.\n\n### Docker image\n\nA docker image can be built from this repo. A prebuilt image with examples\nincluded is also available for download from GitHub Container Registry:\n\n* https://github.com/cloudflare/ebpf_exporter/pkgs/container/ebpf_exporter\n\nTo build the image with just the exporter binary, run the following:\n\n```\ndocker build --tag ebpf_exporter --target ebpf_exporter .\n```\n\nTo run it with the examples, you need to build them first (see above).\nThen you can run by running a privileged container and bind-mounting:\n\n* `$(pwd)/examples:/examples:ro` to allow access to examples on the host\n* `/sys/fs/cgroup:/sys/fs/cgroup:ro` to allow resolving cgroups\n\nYou might have to bind-mount additional directories depending on your needs.\nYou might also not need to bind-mount anything for simple kprobe examples.\n\nThe actual command to run the docker container (from the repo directory):\n\n```\ndocker run --rm -it --privileged -p 9435:9435 \\\n  -v $(pwd)/examples:/examples \\\n  -v /sys/fs/cgroup:/sys/fs/cgroup:ro \\\n  ebpf_exporter --config.dir=examples --config.names=timers\n```\n\nFor production use you would either bind-mount your own config and compiled\nbpf programs corresponding to it, or build your own image based on ours\nwith your own config baked in.\n\nFor development use when you don't want or have any dev tools on the host,\nyou can build the docker image with examples bundled:\n\n```\ndocker build --tag ebpf_exporter --target ebpf_exporter_with_examples .\n```\n\nSome examples then can run without any bind mounts:\n\n```\ndocker run --rm -it --privileged -p 9435:9435 \\\n  ebpf_exporter --config.dir=examples --config.names=timers\n```\n\nOr with the publicly available prebuilt image:\n\n```\ndocker run --rm -it --privileged -p 9435:9435 \\\n  ghcr.io/cloudflare/ebpf_exporter --config.dir=examples --config.names=timers\n```\n\n## Kubernetes Helm chart\n\nA third party helm chart is available here:\n\n* https://github.com/kubeservice-stack/kubservice-charts/tree/master/charts/kubeservice-ebpf-exporter\n\nPlease note that the helm chart is not provided or supported by Cloudflare,\nso do your own due diligence and use it at your own risk.\n\n## Benchmarking overhead\n\nSee [benchmark](benchmark) directory to get an idea of how low ebpf overhead is.\n\n## Required capabilities\n\nWhile you can run `ebpf_exporter` as `root`, it is not strictly necessary.\nOnly the following two capabilities are necessary for normal operation:\n\n* `CAP_BPF`: required for privileged bpf operations and for reading memory\n* `CAP_PERFMON`: required to attach bpf programs to kprobes and tracepoints\n\nIf you are using `systemd`, you can use the following configuration to run\nas on otherwise unprivileged dynamic user with the needed capabilities:\n\n```ini\nDynamicUser=true\nAmbientCapabilities=CAP_BPF CAP_PERFMON\nCapabilityBoundingSet=CAP_BPF CAP_PERFMON\n```\n\nPrior to Linux v5.8 there was no dedicated `CAP_BPF` and `CAP_PERFMON`,\nbut you can use `CAP_SYS_ADMIN` instead of your kernel is older.\n\nIf you pass `--capabilities.keep=none` flag to `ebpf_expoter`, then it drops\nall capabilities after attaching the probes, leaving it fully unprivileged.\n\nThe following additional capabilities might be needed:\n\n* `CAP_SYSLOG`: if you use `ksym` decoder to have access to `/proc/kallsyms`.\n  Note that you must keep this capability: `--capabilities.keep=cap_syslog`.\n  See: https://elixir.bootlin.com/linux/v6.4/source/kernel/kallsyms.c#L982\n* `CAP_IPC_LOCK`: if you use `perf_event_array` for reading from the kernel.\n  Note that you must keep it: `--capabilities.keep=cap_perfmon,cap_ipc_lock`.\n* `CAP_SYS_ADMIN`: if you want BTF information from modules.\n  See: https://github.com/libbpf/libbpf/blob/v1.2.0/src/libbpf.c#L8654-L8666\n  and https://elixir.bootlin.com/linux/v6.5-rc1/source/kernel/bpf/syscall.c#L3789\n* `CAP_NET_ADMIN`: if you use net admin related programs like xdp.\n  See: https://elixir.bootlin.com/linux/v6.4/source/kernel/bpf/syscall.c#L3787\n* `CAP_SYS_RESOURCE`: if you run an older kernel without memcg accounting for\n  bpf memory. Upstream Linux kernel added support for this in v5.11.\n  See: https://github.com/libbpf/libbpf/blob/v1.2.0/src/bpf.c#L98-L106\n* `CAP_DAC_READ_SEARCH`: if you want to use `fanotify` to monitor cgroup changes,\n  which is the preferred way, but only available since Linux v6.6.\n  See: https://github.com/torvalds/linux/commit/0ce7c12e88cf\n\n## External BTF Support\n\nExecution of eBPF programs requires kernel data types normally available\nin `/sys/kernel/btf/vmlinux`, which is created during kernel build process.\nHowever, on some older kernel configurations, this file might not be available.\nIf that's the case, an external BTF file can be supplied with `--btf.path`.\nAn archive of BTFs for all some older distros and kernel versions can be\nfound [here](https://github.com/aquasecurity/btfhub-archive).\n\n## Supported scenarios\n\nCurrently the only supported way of getting data out of the kernel is via maps.\n\nSee [examples](#examples) section for real world examples.\n\nIf you have examples you want to share, please feel free to open a PR.\n\n## Configuration\n\nSkip to [format](#configuration-file-format) to see the full specification.\n\n### Examples\n\nYou can find additional examples in [examples](examples) directory.\n\nUnless otherwise specified, all examples are expected to work on Linux 5.15,\nwhich is the latest LTS release at the time of writing. Thanks to CO-RE,\nexamples are also supposed to work on any modern kernel with BTF enabled.\n\nYou can find the list of supported distros in `libbpf` README:\n\n* https://github.com/libbpf/libbpf#bpf-co-re-compile-once--run-everywhere\n\n#### Building examples\n\nTo build examples, run:\n\n```\nmake -C examples clean build\n```\n\nThis will use `clang` to build examples with `vmlinux.h` we provide\nin this repo (see [include](include/README.md) for more on `vmlinux.h`).\n\nExamples need to be compiled before they can be used.\n\nNote that compiled examples can be used as is on any BTF enabled kernel\nwith no runtime dependencies. Most modern Linux distributions have it enabled.\n\n#### Timers via tracepoints (counters)\n\nThis config attaches to kernel tracepoints for timers subsystem\nand counts timers that fire with breakdown by timer name.\n\nResulting metrics:\n\n```\n# HELP ebpf_exporter_timer_starts_total Timers fired in the kernel\n# TYPE ebpf_exporter_timer_starts_total counter\nebpf_exporter_timer_starts_total{function=\"blk_stat_timer_fn\"} 10\nebpf_exporter_timer_starts_total{function=\"commit_timeout\t[jbd2]\"} 1\nebpf_exporter_timer_starts_total{function=\"delayed_work_timer_fn\"} 25\nebpf_exporter_timer_starts_total{function=\"dev_watchdog\"} 1\nebpf_exporter_timer_starts_total{function=\"mix_interrupt_randomness\"} 3\nebpf_exporter_timer_starts_total{function=\"neigh_timer_handler\"} 1\nebpf_exporter_timer_starts_total{function=\"process_timeout\"} 49\nebpf_exporter_timer_starts_total{function=\"reqsk_timer_handler\"} 2\nebpf_exporter_timer_starts_total{function=\"tcp_delack_timer\"} 5\nebpf_exporter_timer_starts_total{function=\"tcp_keepalive_timer\"} 6\nebpf_exporter_timer_starts_total{function=\"tcp_orphan_update\"} 16\nebpf_exporter_timer_starts_total{function=\"tcp_write_timer\"} 12\nebpf_exporter_timer_starts_total{function=\"tw_timer_handler\"} 1\nebpf_exporter_timer_starts_total{function=\"writeout_period\"} 5\n```\n\nThere's config file for it:\n\n```yaml\nmetrics:\n  counters:\n    - name: timer_starts_total\n      help: Timers fired in the kernel\n      labels:\n        - name: function\n          size: 8\n          decoders:\n            - name: ksym\n```\n\nAnd corresponding C code that compiles into an ELF file with eBPF bytecode:\n\n```C\n#include \u003cvmlinux.h\u003e\n#include \u003cbpf/bpf_tracing.h\u003e\n#include \"maps.bpf.h\"\n\nstruct {\n    __uint(type, BPF_MAP_TYPE_HASH);\n    __uint(max_entries, 1024);\n    __type(key, u64);\n    __type(value, u64);\n} timer_starts_total SEC(\".maps\");\n\nSEC(\"tp_btf/timer_start\")\nint BPF_PROG(timer_start, struct timer_list *timer)\n{\n    u64 function = (u64) timer-\u003efunction;\n    increment_map(\u0026timer_starts_total, \u0026function, 1);\n    return 0;\n}\n\nchar LICENSE[] SEC(\"license\") = \"GPL\";\n```\n\n#### Block IO histograms (histograms)\n\nThis config attaches to block io subsystem and reports disk latency\nas a prometheus histogram, allowing you to compute percentiles.\n\nThe following tools are working with similar concepts:\n\n* https://github.com/iovisor/bcc/blob/master/tools/biosnoop_example.txt\n* https://github.com/iovisor/bcc/blob/master/tools/biolatency_example.txt\n* https://github.com/iovisor/bcc/blob/master/tools/bitesize_example.txt\n\nThis program was the initial reason for the exporter and was heavily\ninfluenced by the experimental exporter from Daniel Swarbrick:\n\n* https://github.com/dswarbrick/ebpf_exporter\n\nResulting metrics:\n\n```\n# HELP ebpf_exporter_bio_latency_seconds Block IO latency histogram\n# TYPE ebpf_exporter_bio_latency_seconds histogram\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"1e-06\"} 0\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"2e-06\"} 0\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"4e-06\"} 0\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"8e-06\"} 0\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"1.6e-05\"} 0\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"3.2e-05\"} 0\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"6.4e-05\"} 0\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"0.000128\"} 22\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"0.000256\"} 36\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"0.000512\"} 40\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"0.001024\"} 48\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"0.002048\"} 48\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"0.004096\"} 48\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"0.008192\"} 48\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"0.016384\"} 48\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"0.032768\"} 48\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"0.065536\"} 48\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"0.131072\"} 48\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"0.262144\"} 48\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"0.524288\"} 48\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"1.048576\"} 48\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"2.097152\"} 48\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"4.194304\"} 48\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"8.388608\"} 48\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"16.777216\"} 48\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"33.554432\"} 48\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"67.108864\"} 48\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"134.217728\"} 48\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme0n1\",operation=\"write\",le=\"+Inf\"} 48\nebpf_exporter_bio_latency_seconds_sum{device=\"nvme0n1\",operation=\"write\"} 0.021772\nebpf_exporter_bio_latency_seconds_count{device=\"nvme0n1\",operation=\"write\"} 48\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"1e-06\"} 0\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"2e-06\"} 0\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"4e-06\"} 0\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"8e-06\"} 0\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"1.6e-05\"} 0\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"3.2e-05\"} 0\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"6.4e-05\"} 0\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"0.000128\"} 0\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"0.000256\"} 0\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"0.000512\"} 0\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"0.001024\"} 1\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"0.002048\"} 1\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"0.004096\"} 1\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"0.008192\"} 1\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"0.016384\"} 1\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"0.032768\"} 1\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"0.065536\"} 1\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"0.131072\"} 1\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"0.262144\"} 1\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"0.524288\"} 1\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"1.048576\"} 1\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"2.097152\"} 1\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"4.194304\"} 1\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"8.388608\"} 1\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"16.777216\"} 1\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"33.554432\"} 1\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"67.108864\"} 1\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"134.217728\"} 1\nebpf_exporter_bio_latency_seconds_bucket{device=\"nvme1n1\",operation=\"write\",le=\"+Inf\"} 1\nebpf_exporter_bio_latency_seconds_sum{device=\"nvme1n1\",operation=\"write\"} 0.0018239999999999999\nebpf_exporter_bio_latency_seconds_count{device=\"nvme1n1\",operation=\"write\"} 1\n```\n\nYou can nicely plot this with Grafana:\n\n![Histogram](./examples/biolatency.png)\n\n## Configuration concepts\n\nThe following concepts exists within `ebpf_exporter`.\n\n### Configs\n\nConfigs describe how to extract metrics from kernel. Each config has\na corresponding eBPF code that runs in kernel to produce these metrics.\n\nMultiple configs can be loaded at the same time.\n\n### Metrics\n\nMetrics define what values we get from eBPF program running in the kernel.\n\n#### Counters\n\nCounters from maps are direct transformations: you pull data out of kernel,\ntransform map keys into sets of labels and export them as prometheus counters.\n\n#### Histograms\n\nHistograms from maps are a bit more complex than counters. Maps in the kernel\ncannot be nested, so we need to pack keys in the kernel and unpack in user space.\n\nWe get from this:\n\n```\nsda, read, 1ms -\u003e 10 ops\nsda, read, 2ms -\u003e 25 ops\nsda, read, 4ms -\u003e 51 ops\n```\n\nTo this:\n\n```\nsda, read -\u003e [1ms -\u003e 10 ops, 2ms -\u003e 25 ops, 4ms -\u003e 51 ops]\n```\n\nPrometheus histograms expect to have all buckets when we report a metric,\nbut the kernel creates keys as events occur, which means we need to backfill\nthe missing data.\n\nThat's why for histogram configuration we have the following keys:\n\n* `bucket_type`: can be either `exp2`, `exp2zero`, `linear`, or `fixed`\n* `bucket_min`: minimum bucket key (`exp2`, `exp2zero` and `linear` only)\n* `bucket_max`: maximum bucket key (`exp2`, `exp2zero` and `linear` only)\n* `bucket_keys`: maximum bucket key (`fixed` only)\n* `bucket_multiplier`: multiplier for bucket keys (default is `1`)\n\n##### `exp2` histograms\n\nFor `exp2` histograms we expect kernel to provide a map with linear keys that\nare log2 of actual values. We then go from `bucket_min` to `bucket_max` in\nuser space and remap keys by exponentiating them:\n\n```\ncount = 0\nfor i = bucket_min; i \u003c bucket_max; i++ {\n  count += map.get(i, 0)\n  result[exp2(i) * bucket_multiplier] = count\n}\n```\n\nHere `map` is the map from the kernel and `result` is what goes to prometheus.\n\nUse `increment_exp2_histogram` in ebpf to observe values.\n\n##### `exp2zero` histograms\n\nThese are the same as `exp2` histograms, except:\n\n* The first key is for the value `0`\n* All other keys are `1` larger than they should be\n\nUse `increment_exp2zero_histogram` in ebpf to observe values.\n\n##### `linear` histograms\n\nFor `linear` histograms we expect kernel to provide a map with linear keys\nthat are results of integer division of original value by `bucket_multiplier`.\nTo reconstruct the histogram in user space we do the following:\n\n```\ncount = 0\nfor i = bucket_min; i \u003c bucket_max; i++ {\n  count += map.get(i, 0)\n  result[i * bucket_multiplier] = count\n}\n```\n\n##### `fixed` histograms\n\nFor `fixed` histograms we expect kernel to provide a map with fixed keys\ndefined by the user.\n\n```\ncount = 0\nfor i = 0; i \u003c len(bucket_keys); i++ {\n  count  += map.get(bucket_keys[i], 0)\n  result[bucket_keys[i] * multiplier] = count\n}\n```\n\n##### `sum` keys\n\nFor `exp2` and `linear` histograms, if `bucket_max + 1` contains a non-zero\nvalue, it will be used as the `sum` key in histogram, providing additional\ninformation and allowing richer metrics.\n\nFor `fixed` histograms, if `buckets_keys[len(bucket_keys) - 1 ] + 1` contains\na non-zero value, it will be used as the `sum` key.\n\n### Labels\n\nLabels transform kernel map keys into prometheus labels.\n\nMaps coming from the kernel are binary encoded. Values are always `u64`, but\nkeys can be either primitive types like `u64` or complex `struct`s.\n\nEach label can be transformed with decoders (see below) according to metric\nconfiguration. Generally the number of labels matches the number of elements\nin the kernel map key.\n\nFor map keys that are represented as `struct`s alignment rules apply:\n\n* `u64` must be aligned at 8 byte boundary\n* `u32` must be aligned at 4 byte boundary\n* `u16` must be aligned at 2 byte boundary\n\nThis means that the following struct:\n\n```c\nstruct disk_latency_key_t {\n    u32 dev;\n    u8 op;\n    u64 slot;\n};\n```\n\nIs represented as:\n\n* 4 byte `dev` integer\n* 1 byte `op` integer\n* 3 byte padding to align `slot`\n* 8 byte `slot` integer\n\nWhen decoding, either specify the padding explicitly with the key `padding` or\ninclude it in the label size:\n\n* 4 for `dev`\n* 4 for `op` (1 byte value + 3 byte padding)\n* 8 byte `slot`\n\n### Decoders\n\nDecoders take a byte slice input of requested length and transform it into\na byte slice representing a string. That byte slice can either be consumed\nby another decoder (for example `string` -\u003e `regexp`) or or used as the final\nlabel value exporter to Prometheus.\n\nBelow are decoders we have built in.\n\n#### `cgroup`\n\nWith cgroup decoder you can turn the u64 from `bpf_get_current_cgroup_id`\ninto a human readable string representing cgroup path, like:\n\n* `/sys/fs/cgroup/system.slice/ssh.service`\n\n#### ifname\n\nIfname decoder takes a network interface index and converts it into its\nname like `eth0`.\n\n#### `dname`\n\nDname decoder read DNS qname from string in wire format, then decode\nit into '.' notation format. Could be used after `string` decoder.\nE.g.: `\\x07example\\03com\\x00` will become `example.com`. This decoder\ncould be used after `string` decode, like the following example:\n\n```yaml\n- name: qname\n  decoders:\n    - name: string\n    - name: dname\n```\n\n#### `errno`\n\nErrno decoder converts `errno` number into a string representation like\n`EPIPE`. It is normally paired with a `unit` decoder as the first step.\n\n### `hex`\n\nHex decoder turns bytes into their hex representation.\n\n#### `inet_ip`\n\nNetwork IP decoded can turn byte encoded IPv4 and IPv6 addresses\nthat kernel operates on into human readable form like `1.1.1.1`.\n\n#### `ksym`\n\nKSym decoder takes kernel address and converts that to the function name.\n\nIn your eBPF program you can use `PT_REGS_IP_CORE(ctx)` to get the address\nof the function you attached to as a `u64` variable. Note that for kprobes\nyou need to wrap it with `KPROBE_REGS_IP_FIX()` from `regs-ip.bpf.h`.\n\n#### `majorminor`\n\nWith major-minor decoder you can turn kernel's combined u32 view\nof major and minor device numbers into a device name in `/dev`.\n\n### `pci_vendor`\n\nWith `pci_vendor` decoder you can transform PCI vendor IDs like 0x8086\ninto human readable vendor names like `Intel Corporation`.\n\n### `pci_device`\n\nWith `pci_vendor` decoder you can transform PCI vendor IDs like 0x80861000\ninto human readable names like `82542 Gigabit Ethernet Controller (Fiber)`.\n\nNote that the you need to concatenate vendor and device id together for this.\n\n### `pci_class`\n\nWith `pci_class` decoder you can transform PCI class ID (the lowest byte) into\nthe class name like `Network controller`.\n\n### `pci_subclass`\n\nWith `pci_subclass` decoder you can transform PCI subclass (two lowest bytes)\ninto the subclass name like `Ethernet controller`.\n\n#### `regexp`\n\nRegexp decoder takes list of strings from `regexp` configuration key\nof the decoder and ties to use each as a pattern in `golang.org/pkg/regexp`:\n\n* https://golang.org/pkg/regexp\n\nIf decoder input matches any of the patterns, it is permitted.\nOtherwise, the whole metric label set is dropped.\n\nAn example to report metrics only for `systemd-journal` and `syslog-ng`:\n\n```yaml\n- name: command\n  decoders:\n    - name: string\n    - name: regexp\n      regexps:\n        - ^(kswapd).*$ # if sub-matches are present, the first one is used for the value\n        - ^systemd-journal$\n        - ^syslog-ng$\n```\n\n#### `static_map`\n\nStatic map decoder takes input and maps it to another value via `static_map`\nconfiguration key of the decoder. Values are expected as strings.\n\nAn example to match `1` to `read` and `2` to `write`:\n\n```yaml\n- name: operation\n  decoders:\n    - name:static_map\n      static_map:\n        1: read\n        2: write\n```\nUnknown keys will be replaced by `\"unknown:key_name\"` unless `allow_unknown: true`\nis specified in the decoder. For example, the above will decode `3` to `unknown:3`\nand the below example will decode `3` to `3`:\n\n```yaml\n- name: operation\n  decoders:\n    - name:static_map\n      allow_unknown: true\n      static_map:\n        1: read\n        2: write\n```\n\n\n#### `string`\n\nString decoder transforms possibly null terminated strings coming\nfrom the kernel into string usable for prometheus metrics.\n\n### `syscall`\n\nSyscall decoder transforms syscall numbers into syscall names.\n\nThe tables can be regenerated by `make syscalls`. See `scripts/mksyscalls`.\n\n#### `uint`\n\nUInt decoder transforms hex encoded `uint` values from the kernel\ninto regular base10 numbers. For example: `0xe -\u003e 14`.\n\n## Per CPU map support\n\nPer CPU map reading is fully supported. If the last decoder for a percpu\nmap is called `cpu` (use 2 byte `uint` decoder), then `cpu` label is\nadded automatically. If it's not present, then the percpu counters are\naggregated into one global counter.\n\nThere is [percpu-softirq](examples/percpu-softirq.bpf.c) in examples.\nSee #226 for examples of different modes of operation for it.\n\n### Configuration file format\n\nConfiguration file is defined like this:\n\n```\n# Metrics attached to the program\n[ metrics: metrics ]\n# Kernel symbol addresses to define as kaddr_{symbol} from /proc/kallsyms (consider CONFIG_KALLSYMS_ALL)\nkaddrs:\n  [ - symbol_to_resolve ]\n```\n\n#### `metrics`\n\nSee [Metrics](#metrics) section for more details.\n\n```\ncounters:\n  [ - counter ]\nhistograms:\n  [ - histogram ]\n```\n\n#### `counter`\n\nSee [Counters](#counters) section for more details.\n\n```\nname: \u003cprometheus counter name\u003e\nhelp: \u003cprometheus metric help\u003e\nperf_event_array: \u003cwhether map is a BPF_MAP_TYPE_PERF_EVENT_ARRAY map: bool\u003e\nflush_interval: \u003chow often should we flush metrics from the perf_event_array: time.Duration\u003e\nlabels:\n  [ - label ]\n```\n\nAn example of `perf_map` can be found [here](examples/oomkill.yaml).\n\n#### `histogram`\n\nSee [Histograms](#histograms) section for more details.\n\n```\nname: \u003cprometheus histogram name\u003e\nhelp: \u003cprometheus metric help\u003e\nbucket_type: \u003cmap bucket type: exp2 or linear\u003e\nbucket_multiplier: \u003cmap bucket multiplier: float64\u003e\nbucket_min: \u003cmin bucket value: int\u003e\nbucket_max: \u003cmax bucket value: int\u003e\nlabels:\n  [ - label ]\n```\n\n#### `label`\n\nSee [Labels](#labels) section for more details.\n\n```\nname: \u003cprometheus label name\u003e\nsize: \u003cfield size\u003e\npadding: \u003cpadding size\u003e\ndecoders:\n  [ - decoder ]\n```\n\n#### `decoder`\n\nSee [Decoders](#decoders) section for more details.\n\n```\nname: \u003cdecoder name\u003e\n# ... decoder specific configuration\n```\n\n## Built-in metrics\n\n### `ebpf_exporter_enabled_configs`\n\nThis gauge reports a timeseries for every loaded config:\n\n```\n# HELP ebpf_exporter_enabled_configs The set of enabled configs\n# TYPE ebpf_exporter_enabled_configs gauge\nebpf_exporter_enabled_configs{name=\"cachestat\"} 1\n```\n\n### `ebpf_exporter_ebpf_program_info`\n\nThis gauge reports information available for every ebpf program:\n\n```\n# HELP ebpf_exporter_ebpf_programs Info about ebpf programs\n# TYPE ebpf_exporter_ebpf_programs gauge\nebpf_exporter_ebpf_program_info{config=\"cachestat\",id=\"545\",program=\"add_to_page_cache_lru\",tag=\"6c007da3187b5b32\"} 1\nebpf_exporter_ebpf_program_info{config=\"cachestat\",id=\"546\",program=\"mark_page_accessed\",tag=\"6c007da3187b5b32\"} 1\nebpf_exporter_ebpf_program_info{config=\"cachestat\",id=\"547\",program=\"folio_account_dirtied\",tag=\"6c007da3187b5b32\"} 1\nebpf_exporter_ebpf_program_info{config=\"cachestat\",id=\"548\",program=\"mark_buffer_dirty\",tag=\"6c007da3187b5b32\"} 1\n```\n\nHere `tag` can be used for tracing and performance analysis with two conditions:\n\n* `net.core.bpf_jit_kallsyms=1` sysctl is set\n* `--kallsyms=/proc/kallsyms` is passed to `perf record`\n\nNewer kernels allow `--kallsyms` to `perf top` as well,\nin the future it may not be required at all:\n\n* https://www.spinics.net/lists/linux-perf-users/msg07216.html\n\n### `ebpf_exporter_ebpf_program_attached`\n\nThis gauge reports whether individual programs were successfully attached.\n\n```\n# HELP ebpf_exporter_ebpf_program_attached Whether a program is attached\n# TYPE ebpf_exporter_ebpf_program_attached gauge\nebpf_exporter_ebpf_program_attached{id=\"247\"} 1\nebpf_exporter_ebpf_program_attached{id=\"248\"} 1\nebpf_exporter_ebpf_program_attached{id=\"249\"} 0\nebpf_exporter_ebpf_program_attached{id=\"250\"} 1\n```\n\nIt needs to be joined by `id` label with `ebpf_exporter_ebpf_program_info`\nto get more information about the program.\n\n### `ebpf_exporter_ebpf_program_run_time_seconds`\n\nThis counter reports how much time individual programs spent running.\n\n```\n# HELP ebpf_exporter_ebpf_program_run_time_seconds How long has the program been executing\n# TYPE ebpf_exporter_ebpf_program_run_time_seconds counter\nebpf_exporter_ebpf_program_run_time_seconds{id=\"247\"} 0\nebpf_exporter_ebpf_program_run_time_seconds{id=\"248\"} 0.001252621\nebpf_exporter_ebpf_program_run_time_seconds{id=\"249\"} 0\nebpf_exporter_ebpf_program_run_time_seconds{id=\"250\"} 3.6668e-05\n```\n\nIt requires `kernel.bpf_stats_enabled` sysctl to be enabled.\n\nIt needs to be joined by `id` label with `ebpf_exporter_ebpf_program_info`\nto get more information about the program.\n\n### `ebpf_exporter_ebpf_program_run_count_total`\n\nThis counter reports how many times individual programs ran.\n\n```\n# HELP ebpf_exporter_ebpf_program_run_count_total How many times has the program been executed\n# TYPE ebpf_exporter_ebpf_program_run_count_total counter\nebpf_exporter_ebpf_program_run_count_total{id=\"247\"} 0\nebpf_exporter_ebpf_program_run_count_total{id=\"248\"} 11336\nebpf_exporter_ebpf_program_run_count_total{id=\"249\"} 0\nebpf_exporter_ebpf_program_run_count_total{id=\"250\"} 69\n```\n\nIt requires `kernel.bpf_stats_enabled` sysctl to be enabled.\n\nIt needs to be joined by `id` label with `ebpf_exporter_ebpf_program_info`\nto get more information about the program.\n\n### `ebpf_exporter_decoder_errors_total`\n\nThis counter reports the number of times labels failed to be decoded by config.\n\n```\n# HELP ebpf_exporter_decoder_errors_total How many times has decoders encountered errors\n# TYPE ebpf_exporter_decoder_errors_total counter\nebpf_exporter_decoder_errors_total{config=\"kstack\"} 0\nebpf_exporter_decoder_errors_total{config=\"sock-trace\"} 4\n```\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudflare%2Febpf_exporter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcloudflare%2Febpf_exporter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudflare%2Febpf_exporter/lists"}