{"id":15151849,"url":"https://github.com/elastic/quark","last_synced_at":"2025-10-19T20:31:26.156Z","repository":{"id":257782037,"uuid":"752347228","full_name":"elastic/quark","owner":"elastic","description":null,"archived":false,"fork":false,"pushed_at":"2025-01-24T15:34:36.000Z","size":14197,"stargazers_count":15,"open_issues_count":16,"forks_count":2,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-01-30T03:51:12.094Z","etag":null,"topics":["bpf","events","kprobe","linux","process","processes","quark","telemetry"],"latest_commit_sha":null,"homepage":"https://elastic.github.io/quark/index.html","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/elastic.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-03T17:14:13.000Z","updated_at":"2025-01-23T16:17:00.000Z","dependencies_parsed_at":"2024-11-12T07:27:54.261Z","dependency_job_id":"036dbd2b-8b80-4bb4-95b1-e64d557e5a7f","html_url":"https://github.com/elastic/quark","commit_stats":{"total_commits":303,"total_committers":3,"mean_commits":101.0,"dds":"0.33003300330033003","last_synced_commit":"fdebd7f3c011d93c71ebcd59bf08ef65ffe63d6f"},"previous_names":["elastic/quark"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Fquark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Fquark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Fquark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Fquark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/elastic","download_url":"https://codeload.github.com/elastic/quark/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237215417,"owners_count":19273548,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bpf","events","kprobe","linux","process","processes","quark","telemetry"],"created_at":"2024-09-26T15:23:03.311Z","updated_at":"2025-10-19T20:31:26.150Z","avatar_url":"https://github.com/elastic.png","language":"C","readme":"# quark — unified system process telemetry library\n\n- [DESCRIPTION](#DESCRIPTION)\n- [QUICKSTART](#QUICKSTART)\n- [FEATURES](#FEATURES)\n- [BUILDING](#BUILDING)\n- [LINKING](#LINKING)\n- [TESTING](#TESTING)\n- [INCLUDED BINARIES](#INCLUDED_BINARIES)\n- [CONVENTIONS](#CONVENTIONS)\n- [BASIC USAGE](#BASIC_USAGE)\n- [EXAMPLES](#EXAMPLES)\n- [API](#API)\n- [FURTHER READING](#FURTHER_READING)\n- [SEE ALSO](#SEE_ALSO)\n- [LICENSE](#LICENSE)\n- [HISTORY](#HISTORY)\n\n# [DESCRIPTION](#DESCRIPTION)\n\nquark is a library that provides a way to retrieve and listen to process events in linux systems. Its main purpose is to abstract different backends and to provide a common API for listening to system-wide events like [fork(2)](https://linux.die.net/man/2/fork), [exec(3)](https://linux.die.net/man/3/exec), [exit(3)](https://linux.die.net/man/3/exit) and others.\n\nquark not only provides an API for listening to events, but also handles ordering, buffering and aggregation of said events. In its most basic form, a short lived process consisting of [fork(2)](https://linux.die.net/man/2/fork) + [exec(3)](https://linux.die.net/man/3/exec) + [exit(3)](https://linux.die.net/man/3/exit) will be aggregated into one `quark_event`. An internal process cache is also kept that can be looked up via [quark\\_process\\_lookup(3)](https://elastic.github.io/quark/quark_process_lookup.3.html).\n\n# [QUICKSTART](#QUICKSTART)\n\nClone the repository, compile and run quark's test utility [quark-mon(8)](https://elastic.github.io/quark/quark-mon.8.html):\n\n```\n$ git clone --recursive https://github.com/elastic/quark\n$ cd quark\n$ make\n$ sudo ./quark-mon\n\nOn another shell, create any process like:\n$ ls -1 /tmp | wc -l\n```\n\nSee [BUILDING](#BUILDING) for a list of dependencies if you're having trouble building. Also see [INCLUDED BINARIES](#INCLUDED_BINARIES) and [quark-mon(8)](https://elastic.github.io/quark/quark-mon.8.html).\n\n# [FEATURES](#FEATURES)\n\n[*ORDERING*](#ORDERING)\n\nquark tries to guarantee event ordering as much as possible. Ordering must be done in userland for some backends, notably anything that uses perf-rings. quark uses two [*Rank Balanced Trees*](#Rank) for ordering and aggregation.\n\nThe first tree is basically a priority queue, ordered by the time of the event. The second tree is ordered by time of the event + pid and it's used for event aggregation.\n\n[*AGGREGATION*](#AGGREGATION)\n\nquark buffers and aggregates related events that happened close enough. The common case is generating a single event for the triple: [fork(2)](https://linux.die.net/man/2/fork), [exec(3)](https://linux.die.net/man/3/exec), [exit(3)](https://linux.die.net/man/3/exit). There are rules on what can be aggregated, and only events of the same pid are aggregated. For example: quark won't aggregate two [exec(3)](https://linux.die.net/man/3/exec) events, otherwise we would lose the effects of the first one. These rules will be exposed and configurable in the future.\n\n[*BUFFERING*](#BUFFERING)\n\nFor aggregation and ordering to work, quark needs to be able to buffer events, this means holding them before presenting them to the user. quark employs an ageing timeout that is a stepped function of the number of currently buffered events, the more events you have, the shorter the timeout will be, so memory can be bound. A `quark_event` is only given to the user when it has a certain age. From quark.c:\n\n```\n/*\n * Target age is the duration in ns of how long should we hold the event in the\n * tree before processing it. It's a function of the number of items in the tree\n * and its maximum capacity:\n * from [0; 10%]    -\u003e 1000ms\n * from [90%; 100%] -\u003e 0ms\n * from (10%; 90%)  -\u003e linear from 1000ms -\u003e 100ms\n */\n```\n\n[*ENRICHMENT*](#ENRICHMENT)\n\nThe library tries to give as much context for an event as possible. Depending on the backend, the events we read from the kernel can be limited in context. quark maintains an internal process table with what has been learned about the process so far, this context is then included in each event given to the user. The process table can also be queried, see below.\n\n[*PROCESS CACHE*](#PROCESS)\n\nAn internal cache of processes is kept that can be looked up via [quark\\_process\\_lookup(3)](https://elastic.github.io/quark/quark_process_lookup.3.html). This cache keeps soon-to-be-purged elements for a little while so that you can still lookup a process that just exited. The table is initialized by scraping /proc.\n\n[*TRANSPARENCY*](#TRANSPARENCY)\n\nquark tries to be as transparent as possible about what it knows, there are counters for lost events, and each piece of information of a `quark_event` is guarded by a flag, meaning the user might get incomplete events in the case of lost events, it's the user responsability to decide what to do with it.\n\nDepending on load, the user might see an event as the aggregation of multiple events, or as independent events. The content remains the same.\n\n[*LANGUAGE BINDINGS*](#LANGUAGE)\n\nquark is written in C, but Go bindings are also provided. Ideally we will be able to provide bindings for other languages in the future.\n\n[*MULTIPLE BACKENDS*](#MULTIPLE)\n\nCurrently, EBPF and a kprobe-based backend are provided, but we would like to add AUDIT support as well. The backend in use is transparent to the user and unless specified, quark will try to use the EBPF, falling back to KPROBE if it failed.\n\n# [BUILDING](#BUILDING)\n\nquark can be built natively or via a container, native is preferred and depends on:\n\n- bpftool\n- clang\n- gnumake\n- gcc\n- mandoc (for docs)\n- html2markdown utility  \n  (for docs, pre-built binaries are available at  \n  https://github.com/JohannesKaufmann/html-to-markdown/releases)\n- m4\n- qemu (for testing)\n- zstd\n\nMake sure to clone the repository recursively: [*git clone --recursive*](#git).\n\n*make* builds the repository, including quark-mon, libquark\\_big.a and a libquark.a.\n\nlibquark\\_big.a includes all needed dependencies in one big archive. This includes a libbpf.a, libelf\\_pic.a (from the elftoolchain project, BSD license), and a libz.a (see zlib/LICENSE). See [LINKING](#LINKING) to learn how to link either.\n\nWhile quark doesn't build *elastic/ebpf*, it does use the EBPF programs from that repository, only the files needed are included in quark, as *elastic/ebpf* is quite big.\n\nOther useful build targets include:\n\n[*clean*](#clean)\n\nClean object files from quark.\n\n[*docker*](#docker)\n\nBuilds quark inside a docker container, so you don't have to worry about having build dependencies.\n\n[*docker-shell*](#docker-shell)\n\nSpawns an interactive shell inside the same builder container created by ‘make docker’. Handy for debugging failed builds, inspecting artifacts etc.\n\n[*docker-cross-arm64*](#docker-cross-arm64)\n\nBuilds quark for arm64 inside a docker container.\n\n[*centos7*](#centos7)\n\nBuilds quark inside a centos7 docker container, useful for linking against ancient glibc-2.17.\n\n[*centos7-shell*](#centos7-shell)\n\nOpens an interactive shell in the centos7 builder container.\n\n[*alpine*](#alpine)\n\nBuilds quark inside an Alpine Linux docker container, so we can track musl builds.\n\n[*alpine-shell*](#alpine-shell)\n\nInteractive shell inside the Alpine builder image.\n\n[*test*](#test)\n\nBuilds and runs [quark-test(8)](https://elastic.github.io/quark/quark-test.8.html).\n\n[*test-valgrind*](#test-valgrind)\n\nBuilds and runs [quark-test(8)](https://elastic.github.io/quark/quark-test.8.html) under valgrind.\n\n[*test-kernel*](#test-kernel)\n\nRuns [quark-test(8)](https://elastic.github.io/quark/quark-test.8.html) over all kernels in kernel\\_images/.\n\n[*btfhub*](#btfhub)\n\nRegenerates btfhub.c. Usage:\n\n```\n$ make btfhub BTFHUB_ARCHIVE_PATH=/my/path/to/btfhub-archive\n```\n\n[*clean-all*](#clean-all)\n\nClean all object files, including the ones from [*libbpf*](#libbpf), [*libz*](#libz) and [*libelf*](#libelf).\n\n[*docs*](#docs)\n\nLints and generates all the documentation from manpages in docs/.\n\n[*svg*](#svg)\n\nBuilds an SVG out of the DOT files produced by [quark-mon(8)](https://elastic.github.io/quark/quark-mon.8.html).\n\n[*README.md*](#README.md)\n\nGenerates README.md out of quark.7.\n\n[*eebpf-sync*](#eebpf-sync)\n\nCopies the files from EEBPF\\_PATH used by quark. Usage:\n\n```\n$ make eebpf-sync EEBPF_PATH=/my/path/to/elastic/ebpf\n```\n\n[*initramfs.gz*](#initramfs.gz)\n\nBuilds an initramfs file containing all quark binaries so that it can be run as the init process on boot, useful for testing any kernel under qemu. See [TESTING](#TESTING).\n\nAll the targets above can generate debug output by specifying [*V=1*](#V=1), as in:\n\n```\n$ make V=1\n```\n\n# [LINKING](#LINKING)\n\n```\n$ cc -o myprogram myprogram.c libquark_big.a\nOR\n$ cc -o myprogram myprogram.c libquark.a libbpf/src/libbpf.a elftoolchain/libelf/libelf_pic.a zlib/libz.a\n```\n\n# [TESTING](#TESTING)\n\n[quark-test(8)](https://elastic.github.io/quark/quark-test.8.html) is the main test utility ran by the CI, can be invoked via *make test*. All tests are self-contained in this binary.\n\nSome included kernels can be tested in qemu via *make test-kernel*. Any quark utility can be run on a custom kernel via the krun.sh script, as in:\n\n```\n$ make initramfs.gz\n$ ./krun.sh initramfs.gz kernel-images/amd64/linux-4.18.0-553.el8_10.x86_64 quark-test -vvv\n```\n\nConvenience wrappers for Fedora, RHEL, and Ubuntu, automate the above by fetching the appropriate kernel packages, extracting vmlinuz and boot-strapping qemu-system-x86\\_64:\n\n```\n$ make initramfs.gz\n$ ./krun-fedora.sh initramfs.gz 40 quark-test -vvv\n$ ./krun-rhel.sh -v initramfs.gz 9 quark-test\n$ ./krun-ubuntu.sh initramfs.gz 24.04 quark-test -b t_dns\n```\n\nThe version number after ‘initramfs.gz’ selects the Fedora, RHEL, or Ubuntu version. All remaining arguments are passed verbatim to [quark-test(8)](https://elastic.github.io/quark/quark-test.8.html), enabling targeted runs such as:\n\n```\n$ ./krun-fedora.sh initramfs.gz 41 quark-test -b t_fork_exec_exit\n```\n\nThese scripts require KVM access and therefore must be executed on a host kernel as root. They are unsuitable for container environments; the docker targets only build quark and do not attempt to run the test suite.\n\n*make test-valgrind* runs the same suite under valgrind and is useful for catching memory errors, while *make test-kernel* cycles through a set of kernel images in kernel\\_images folder to ensure probe compatibility.\n\n# [INCLUDED BINARIES](#INCLUDED_BINARIES)\n\n[quark-mon(8)](https://elastic.github.io/quark/quark-mon.8.html) is a program that dumps `quark_events` to stdout and can be used for demo and debugging. It has a neat feature: can be run without priviledges, while useless in this small program, it aims to demonstrate how a user could implement the same.\n\n[quark-btf(8)](https://elastic.github.io/quark/quark-btf.8.html) is a program for dumping BTF information used by quark.\n\n[quark-test(8)](https://elastic.github.io/quark/quark-test.8.html) is a program for running tests during development.\n\n# [CONVENTIONS](#CONVENTIONS)\n\n- Library calls fail with -1 unless otherwise stated, and `errno` is set.\n- Quark returns pointers to internal state, which must not be modified and/or stored. In the case of multithreading, these pointers should not be accessed if another thread is driving quark through [quark\\_queue\\_get\\_event(3)](https://elastic.github.io/quark/quark_queue_get_event.3.html).\n- No threads are created, the library is driven solely through [quark\\_queue\\_get\\_event(3)](https://elastic.github.io/quark/quark_queue_get_event.3.html).\n- Access to a `quark_queue` must be synchronized by the user in the case of multithreading.\n\n# [BASIC USAGE](#BASIC_USAGE)\n\nThe ball starts with [quark\\_queue\\_open(3)](https://elastic.github.io/quark/quark_queue_open.3.html).\n\n[quark\\_queue\\_open(3)](https://elastic.github.io/quark/quark_queue_open.3.html) initializes a `quark_queue` which holds the majority of runtime state used by library, this includes perf-rings, file descriptors, EBPF programs buffering data-structures and the like. It must be paired with a [quark\\_queue\\_close(3)](https://elastic.github.io/quark/quark_queue_close.3.html) on exit.\n\n[quark\\_queue\\_get\\_event(3)](https://elastic.github.io/quark/quark_queue_get_event.3.html) is the main driver of the library, it does the buffering, per-ring scanning, aggregation and event cache garbage collection. In case there are no events it returns NULL and the user is expected to call [quark\\_queue\\_block(3)](https://elastic.github.io/quark/quark_queue_block.3.html) or equivalent.\n\n# [EXAMPLES](#EXAMPLES)\n\n```\n#include \u003cerr.h\u003e\n#include \u003cquark.h\u003e\n#include \u003cstdio.h\u003e\n\nint\nmain(void)\n{\n\tstruct quark_queue\t \t qq;\n\tconst struct quark_event\t*qev;\n\n\tif (quark_queue_open(\u0026qq, NULL) == -1)\n\t\terr(1, \"quark_queue_open\");\n\n\tfor (; ;) {\n\t\tqev = quark_queue_get_event(\u0026qq);\n\n\t\t/* No events, just block */\n\t\tif (qev == NULL) {\n\t\t\tquark_queue_block(qq);\n\t\t\tcontinue;\n\t\t}\n\n\t\tquark_event_dump(qev, stdout);\n\t}\n\n\tquark_queue_close(\u0026qq);\n\n\treturn (1);\n}\n```\n\n# [API](#API)\n\n[quark\\_queue\\_open(3)](https://elastic.github.io/quark/quark_queue_open.3.html)\n\nopen a queue to receive events, initial library call.\n\n[quark\\_queue\\_default\\_attr(3)](https://elastic.github.io/quark/quark_queue_default_attr.3.html)\n\nget default attributes of [quark\\_queue\\_open(3)](https://elastic.github.io/quark/quark_queue_open.3.html).\n\n[quark\\_queue\\_get\\_event(3)](https://elastic.github.io/quark/quark_queue_get_event.3.html)\n\nget event, main library call.\n\n[quark\\_process\\_lookup(3)](https://elastic.github.io/quark/quark_process_lookup.3.html)\n\nlookup a process in quark's internal cache\n\n[quark\\_event\\_dump(3)](https://elastic.github.io/quark/quark_event_dump.3.html)\n\ndump event, mainly a debugging utility.\n\n[quark\\_process\\_iter(3)](https://elastic.github.io/quark/quark_process_iter.3.html)\n\niterate over existing processes.\n\n[quark\\_queue\\_get\\_epollfd(3)](https://elastic.github.io/quark/quark_queue_get_epollfd.3.html)\n\nget a descriptor suitable for blocking.\n\n[quark\\_queue\\_block(3)](https://elastic.github.io/quark/quark_queue_block.3.html)\n\nblock for an unspecified amount of time.\n\n[quark\\_queue\\_get\\_stats(3)](https://elastic.github.io/quark/quark_queue_get_stats.3.html)\n\nbasic queue statistics.\n\n[quark\\_queue\\_close(3)](https://elastic.github.io/quark/quark_queue_close.3.html)\n\nclose a queue.\n\n# [FURTHER READING](#FURTHER_READING)\n\n[quark\\_queue\\_get\\_event(3)](https://elastic.github.io/quark/quark_queue_get_event.3.html) is the meat of the library and contains further useful documentation.\n\n[quark-mon(8)](https://elastic.github.io/quark/quark-mon.8.html) is the easiest way to get started with quark.\n\n[quark\\_queue\\_open(3)](https://elastic.github.io/quark/quark_queue_open.3.html) describes initialization options that can be useful.\n\n# [SEE ALSO](#SEE_ALSO)\n\n[quark\\_event\\_dump(3)](https://elastic.github.io/quark/quark_event_dump.3.html), [quark\\_process\\_iter(3)](https://elastic.github.io/quark/quark_process_iter.3.html), [quark\\_process\\_lookup(3)](https://elastic.github.io/quark/quark_process_lookup.3.html), [quark\\_queue\\_block(3)](https://elastic.github.io/quark/quark_queue_block.3.html), [quark\\_queue\\_close(3)](https://elastic.github.io/quark/quark_queue_close.3.html), [quark\\_queue\\_get\\_epollfd(3)](https://elastic.github.io/quark/quark_queue_get_epollfd.3.html), [quark\\_queue\\_get\\_event(3)](https://elastic.github.io/quark/quark_queue_get_event.3.html), [quark\\_queue\\_get\\_stats(3)](https://elastic.github.io/quark/quark_queue_get_stats.3.html), [quark\\_queue\\_open(3)](https://elastic.github.io/quark/quark_queue_open.3.html), [quark-btf(8)](https://elastic.github.io/quark/quark-btf.8.html), [quark-mon(8)](https://elastic.github.io/quark/quark-mon.8.html), [quark-test(8)](https://elastic.github.io/quark/quark-test.8.html)\n\n# [LICENSE](#LICENSE)\n\nquark is released under the Apache-2.0 license and contains code under BSD-2, BSD-3, ISC, and zlib Licenses.\n\n# [HISTORY](#HISTORY)\n\nquark started in April 2024.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felastic%2Fquark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Felastic%2Fquark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felastic%2Fquark/lists"}