{"id":18555243,"url":"https://github.com/c3d/recorder","last_synced_at":"2025-04-09T23:32:10.164Z","repository":{"id":19419337,"uuid":"86998901","full_name":"c3d/recorder","owner":"c3d","description":"A lock-free real-time flight recorder for your C or C++ programs","archived":false,"fork":false,"pushed_at":"2024-02-04T23:44:42.000Z","size":778,"stargazers_count":25,"open_issues_count":2,"forks_count":7,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-24T13:44:07.110Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/c3d.png","metadata":{"files":{"readme":"README.md","changelog":"NEWS","contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-04-02T16:21:09.000Z","updated_at":"2024-12-20T03:15:05.000Z","dependencies_parsed_at":"2024-02-05T00:42:47.900Z","dependency_job_id":null,"html_url":"https://github.com/c3d/recorder","commit_stats":null,"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c3d%2Frecorder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c3d%2Frecorder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c3d%2Frecorder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c3d%2Frecorder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/c3d","download_url":"https://codeload.github.com/c3d/recorder/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248129902,"owners_count":21052657,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T21:25:46.070Z","updated_at":"2025-04-09T23:32:08.266Z","avatar_url":"https://github.com/c3d.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# recorder\nA lock-free, real-time flight recorder for your C or C++ programs\n\n* Records information about your program continuously while it's\n  running using simple `printf`-like statements\n* Classify recorded data in different categories to preserve both old\n  important events and recent, rapidly-firing events\n* Dump recorded data on demand, notably in response to signals or from debugger\n* Can trace specific categories of records, i.e. print them as they happen\n* Can export specific data to shared memory channels for external visualization\n* A simple Qt5.9-based visualization tool shows content of data\n  channels in real-time\n\n[![Flight Recorder Presentation Video](http://img.youtube.com/vi/kEnQY1zFa0Y/0.jpg)](http://www.youtube.com/watch?v=kEnQY1zFa0Y \"Flight Recorder Presentation\")\n\n## Instrumentation that is always there when you need it\n\nThe flight recorder is designed to help you debug complex, real-time,\nmulti-CPU programs. It lets you instrument their execution with\nnon-intrusive `printf`-like *record statements*, that capture what is\nhappening in your program. Here is what a `RECORD` statement looks like:\n\n    RECORD(main, \"Program %s started with %d arguments\", argv[0], argc);\n\nIn that example, `main` is the name of the recorder holding the\ndata. We can declare it to hold 32 entries with a declaration like:\n\n    RECORDER(main, 32, \"This is 'main', used in the example above\");\n\nThese `RECORD` statements are very inexpensive (less than 0.1\nmicrosecond on a modern PC), so you can leave them in your code all\nthe time, even for optimized code. See *Performance considerations* at\nend of this document for details.\n\nThe `RECORD` can also be written in lowercase, as `record`, if you\ndon't want macros standing out in your regular code. Having two\nspellings makes it possible to `#undef` either `RECORD` or `record` in\ncase of conflict with existing names in your source code.\n\n    record(main, \"Program %s started with %d arguments\", argv[0], argc);\n\nWhen something bad happens or from within a debugger, you can\n*dump the recorder* by calling the `recorder_dump()` function. This\ngives you a very detailed account of recent events, helping you figure\nout how you came there. The recorder dump also contains highly\ndetailed ordering and timing information, which can be very precious\nin multi-CPU systems.\n\nMultiple recorders can be active simultaneously, for example to\ncapture events from different subsystems. When a recorder dump occurs,\nevents from different recorders are sorted so that you get a picture\nof how the systems interacted. Different recorders can have different\nsize and different refresh rates, but a very fast recorder will not\npush away data from a slower one. This ensures that you can record\nimportant, but slow, events, as well as much more frequent ones.\n\nHere is what a recorder dump can look like (lines omitted for brevity):\n\n    recorder.c:518: [0 0.000000] signals: Activating dump for signal mask 0xE3001C58\n    recorder_test.c:126: [1 0.000008] MAIN: Launching 16 normal recorder threads\n    recorder_test.c:128: [2 0.000045] MAIN: Starting normal speed test for 10s with 16 threads\n    recorder_test.c:137: [2392 0.000230] MAIN: Normal recorder testing in progress, please wait about 10s\n    recorder_test.c:141: [198750457 9.999914] MAIN: Normal recorder testing completed, stopping threads\n    recorder_test.c:98: [198750757 9.999929] SpeedTest: [thread 3] Recording 12448096\n    recorder_test.c:98: [198750760 9.999955] SpeedTest: [thread 7] Recording 12527067\n    recorder_test.c:147: [198750785 9.999930] Pauses: Waiting for recorder threads to stop, 16 remaining\n    recorder_test.c:89: [198750787 9.999946] Pauses: Pausing #0 2401.618us\n    recorder_test.c:98: [198750789 10.000008] SpeedTest: [thread 11] Recording 12385726\n    recorder_test.c:98: [198750763 9.999929] SpeedTest: [thread 3] Recording 12448097\n    recorder_test.c:98: [198750788 9.999971] SpeedTest: [thread 12] Recording 12356775\n    recorder_test.c:98: [198750790 10.000029] SpeedTest: [thread 4] Recording 12412805\n    recorder_test.c:151: [198750791 10.003075] MAIN: Normal test: all threads have stopped, 198750784 iterations\n    recorder_test.c:165: [198750792 10.003117] MAIN: Recorder test complete (Normal version), 16 threads.\n    recorder_test.c:166: [198750793 10.003119] MAIN:   Iterations      =  198750784\n    recorder_test.c:167: [198750794 10.003121] MAIN:   Iterations / ms =      19875\n    recorder_test.c:169: [198750795 10.003122] MAIN:   Record cost     =         50ns\n\nLines begin with the source code location in the program where the\nrecord was taken. This is optional, and is under control of a recorder\ntweak called `recorder_location`. Another tweak, `recorder_functions`,\nselects whether that location includes the function name.\n\nThe number following the source location is the *order* of records, a\nglobal sequence number that helps relate records made in different\nrecorders or from different threads. The display of the order can be\nconfigured using the `recorder_order` tweak.\n\nThe order is followed by a *timestamp*, which counds the number of\nseconds since the start of the program. On 32-bit machines, the\ntimestamp is precise to the ms. On 64-bit machines, it is precise to\nthe microsecond. The time stamp can be displayed as absolute time\nsince midnight using the `recorder_abstime` tweak. The relative\ntime can be disabled by setting `recorder_reltime` tweak to 0.\nWith `recorder_abstime=1 recorder_reltime=0`, the recorder dump will\nlook as follows:\n\n     [11 09:06:16.718056] recorder: Activating tweak 'recorder_abstime' (0x10cba0910)\n     [12 09:06:16.718062] recorder: Activating 'MAIN' (0x10cb84148)\n     [13 09:06:16.718062] recorder: Activating 'Pauses' (0x10cb851b8)\n     [14 09:06:16.718063] recorder: Activating 'Special' (0x10cb89228)\n     [15 09:06:16.718063] recorder: Activating 'SpeedTest' (0x10cb8a298)\n\nFinally, the rest of the record is a printout of what was recorded.\n\nThe recorder dump is generally sorted according to order and should\nshow time stamps in increasing order. However, as the example above\nshows, this may not always be the case. See *Multithreading\nconsiderations* below for an explanation of what this means.\n\nSee [this blog article](https://grenouillebouillie.wordpress.com/2016/12/09/a-real-time-lock-free-multi-cpu-flight-recorder)\nfor a more extensive description of the design and rationale.\n\n\n## Building the recorder library\n\nTo build and test the recorder library on your system, type:\n\n    make test\n\nThis should build the library itself, which really consists of two\nheaders (`recorder.h` and `recorder_ring.h`) and corresponding C files\n(`recorder.c` and `recorder_ring.c`). After building, it will run a\nfew simple tests that perform some operations and record what is\nhappening while they do so.\n\n\n## Adding recorders to your own project\n\nIn order to add recorders to your own C project, you need to integrate\nfour source files:\n\n* The `recorder.h` file is the header, which is designed to work for\n  either C programs. That header relies on a supporting `recorder_ring.h`\n  header to implement ring buffers.\n\n* The `recorder.c` file is the implementation file, which provides\n  support for C programs. The `recorder_ring.c` file implements functions\n  implementing dynamic ring buffers.\n\nTo define recorders, you use `RECORDER` statements, which takes\nthree arguments: the name of the recorder, the number of entries to\nkeep, and a description. You can look at the `hanoi_test.c` file for\nan example of use.  This example defines for example a recorder called\n`MOVES` with 1024 entries, declared as follows:\n\n    RECORDER(MOVES, 1024, \"Moving pieces around\")\n\nIt is also possible to declare recorders in a header file using the\n`RECORDER_DECLARE` statement that takes the name of the recorder.\n\n    RECORDER_DECLARE(MOVES)\n\nThis makes it possible to share a recorder across multiple C source\nfiles. In short, you would typically put `RECORDER` lines in C source\nfiles, and `RECORDER_DECLARE` statements in C header files.\n\n\n## Recording events\n\nTo record events in C, you use a `printf`-like `RECORD` statement,\nwhich begins with the name of the recorder as declared with\n`RECORDER_DECLARE` or defined with `RECORDER`. The `RECORD` statement\nworks mostly like `printf`, but takes a first argument specifying the\nname of the recorder:\n\n    record(MOVES, \"Move disk from %s to %s\\n\", name[left], name[right]);\n\nThe trailing `\\n` in the format string is optional. At recorder dump\ntime, separate records will always be printed on separate lines.\n\nWhile a `RECORD` behaves mostly like `printf`, there are important\ncaveats and limitations to be aware of, as well as interesting format string\nextensions relative to `printf`. Notably:\n\n* The `%+s` format indicates a string that is safe to print even at dump\n  time, for example a static string constant.\n\n* You can [define custom formats](#custom-recorder-data-types) such as `%t`.\n\n* Putting a `\u003e`, `\u003c` or `=` sign as the first character of the format string\n  will indent, unindent or reset the indent. The character is not printed. This\n  is useful to keep track of nested algorithms.\n\n\n## Caveats and limitations\n\nEach `RECORD` statement can have only up to 12 arguments.\nEach individual record can store up to 4 arguments, but the `RECORD`\nmacro can generate 1, 2 or 3 recorder entries depending on the number\nof arguments. If you need more, the changes to the code should be\nsomewhat straightforward.\n\nYou can pass integer values, floating-point values (limited to `float`\non 32-bit machines for size reasons), pointers or strings as `RECORD`\nargument.\n\nHowever, unlike `printf`, the rendering of the final message is done\n*at the time of the dump*. This is not a problem for integer,\ncharacters, pointer or floating-point values, but for strings (the\n`%s` format of `printf`), this could cause a crash. Therefore, a\nstring is printed as a pointer at dump time, unless you add the `+`\nmodifier to the `printf` format, e.g. `\"%+s\"`. If you know that the\nstring is a compiler constant, you can use this flag safely.\n\n    // OK if 0 \u003c= i and i \u003c 5, can use the '%+s' format\n    const char *array[5] = { \"ONE\", \"TWO\", \"THREE\", \"FOUR\", \"FIVE\" };\n    record(Main, \"Looking at %+s\", array[i]);\n\n    // Not OK because the value of the string has been freed at dump time\n    char *tempStr = strdup(\"Hello\");\n    record(Main, \"You will see a pointer at dump time here: %s\", tempStr);\n    free(tempStr); // At dump time, the string no longer exists\n\nNote that if tracing is enabled for a given recorder, the string will be\nprinted. If the string is not valid at that time, a crash is possible\nduring tracing.\n\nThe `RECORD` macro automatically converts floating point values\nto `uintptr_t` based on their type, i.e. 32-bit or 64-bit floating-point.\n\nThe value of the current indent can be read using `recorder_indent()`, but only\nduring tracing or dumping. This can be useful if you want to indent the output\nof your own extended tracing.\n\n\n## Dumping recorder events\n\nTo dump recorded events, you use the `recorder_dump` function. This\ndumps all the recorders:\n\n    recorder_dump();\n\nThis can be used in a number of situations, in particular from a\ndebugger. In `gdb` for example, you could run the following command to\ndump the recorder during a debugging session:\n\n    p recorder_dump()\n\nIf you want to dump specific recorders, you can use\n`recorder_dump_for`, which matches the recorder names against the\nregular expression given as an argument:\n\n    recorder_dump_for(\".*TIMING.*\"); // Dumps \"TIMING\", but also \"MY_TIMING_2\"\n\nDuring a dump, events are sorted according to the global event order.\nNote that sorting only happens across recorders. Within a single\nrecorder, events may be out of order, see *Multithreading\nconsiderations* below.\n\nThe function `recorder_background_dump(pattern)` launches a background\nthread that dumps the recorders selected by `pattern` at regular\ninterval. The sleep time in milliseconds between recorder dumps is\nconfigured by a recorder tweak named `recorder_dump_sleep`, which\ndefaults to 100 ms. The background dump can be stopped by calling the\n`recorder_background_stop` function.\n\nRecorder output normally goes to standard error `stderr`, although it\nis possible to redirect the output using one of the following methods:\n\n* By using the `@output` command, for example from the\n `RECORDER_TRACES` environment variable if your applciation supports\n  it,\n\n* By passing a `FILE *` to `recorder_configure_output` in your\n  application,\n\n* By replacing the output function entirely using `recorder_configure_show`,\n  in which case this function is free to interpret the recorder output\n  pointer in any way you wish. Note that doing so disables the\n  `@output` command.\n\nThe recorder output can be configured with a number of tweaks, including:\n\n* Set `recorder_location` to show the location in the source code (file, line)\n* Set `recorder_function` to show the name of the function\n* Set `recorder_order` to show the order of each recorder entry\n* Set `recorder_abstime` to show the absolute time since start of program\n* Set `recorder_reltime` to show the relative time between two entries\n* Set `recorder_indent` to the maximum indent to use when dumping the recorder\n\n\n## Recorder tracing\n\nIn some cases, it is useful to print specific information as you go\ninstead of after the fact. In this flight recorder, this is called\n*tracing*, and can be enabled for each recorder individually.\n\nWhen tracing for a given recorder is enabled, the trace entries for\nthat recorder are dumped immediately after being recorded. They are\nstill stored in the flight recorder for later replay by\n`recorder_dump`.\n\nTracing can be activated by the `recorder_trace_set` function, which\ntakes a string specifying which traces to activate. The specification\nis a colon or space separated list of trace settings, each of them\nspecifying a regular expression to match recorder names, optionally\nfollowed by an `=` sign and a numerical value.\n\nIf no numerical value is given, then the value `1` is assumed by default, which\nenables tracing. If the trace name begins with `-`, then the value `0` is\nassumed by default, which disables tracing, but preserves recording.\nIf the trace name begins wiht `/`, then the value `-1` is assumed by default,\nwhich disables even recording.\n\nFor example, the trace specification `foo:bar=0:b[a-z]z.*=3:-glop` sets the\nrecorder trace for `foo` to value `1` (enabling tracing for that recorder), sets\nthe recorder trace for `bar` and `glop` to `0`, and sets all recorders with a\nname matching regular expression `b[a-z]z.*` to value `3`, for example `boz` and\n`bbz`.\n\nThe following names in a trace specification denote *command* which\nperform specific actions.\n\n* The `list` and `help` commands will print the list of available\n  recorders on `stderr`.\n\n* The `all` value will be turned into the catch-all `.*`\n  regular expression.\n\n* The `output` command will select a file as output for the recorder,\n  and the `output_append` variant will append to the given file. For\n  example, you can write to `/my/file` using `output=/my/file`.\n\n* The `share` value can be used to set the file name used for sharing\n  information in real-time between a recorder appplication and an\n  application using that data. Note that another way to achieve the\n  same objective is to use the `RECORDER_SHARE` environment variable.\n\n* THe `dump` command causes a recorder dump. This is mostly useful\n  over a remote control session, see `recorder_scope` below.\n\nIf the name of a command conflicts with the name of a recorder, the\nrecorder will be selected, and you can force the selection of the\ncommand by prefixing it with `@`. For example, if a recorder is called\n`output`, you can still output to a file with `@output=/my/file`.\n\n## Using the `RECORDER_TRACES` and `RECORDER_DUMP` environment variables\n\nIf your application calls `recorder_dump_on_common_signals` (see below),\n then traces will be activated or deactivated according to the\n`RECORDER_TRACES` environment variable, and background dump can be\n activated according to the `RECORDER_DUMP` environment variable.\n\nYour application can define traces from another application-specific\nenvironment variable with code like:\n\n    recorder_trace_set(getenv(\"MY_APP_TRACES\"));\n\n\n## Recorder channels\n\nIf the specification of the trace is non-numerical, then it defines a\ncomma-separated list of names of *exported channels*. Data from\nexported channels is made available to an external application,\ne.g. for real-time display using the `recorder_scope` application.\n\nFor example, consider the following `RECORD` statement, taken from the\n`recorder_test.c` example:\n\n    record(SpeedInfo, \"Iterations per millisecond: %lu (%f ns)\",\n           k - last_k, 1e6 / (k - last_k));\n\nThis `RECORD` statements stores two data elements in the `SpeedInfo`\nrecorder each time it is executed, a long unsigned value, and a\nfloating-point value.\n\nIt is then possible to cause to export two data channels holding the\nvalues being recorded, under then names `iter` and `duration`, by\nusing a trace specification such as `SpeedInfo=iter,duration`. For\nexampe, you could run the `recorder_test` program as follows (the\n`KEEP_RUNNING` environment variable being used to keep the program\nrunning for a long time):\n\n    export KEEP_RUNNING=1\n    export RECORDER_TRACES='SpeedInfo=iter,duration'\n    export RECORDER_SHARE=/tmp/recorder_share\n    ./build/objects/linux/opt/recorder_test 1 1\n\nThe data is exported in file `/tmp/recorder_share`. Another program\ncan be used to display the data exported in that file in\nreal-time. The `recorder_scope` program in subdirectory `scope` is an\nexample of such a program. You would run it as follows:\n\n    export RECORDER_SHARE=/tmp/recorder_share\n    ./recorder_scope iter duration\n\nThe result should be something like the picture below:\n\n![Recorder scope](scope/Scope.png)\n\nWhile multiple `RECORD` statements can write to the same recorder, if\nyou intend to export data to channels, the data being exported must\nbe positionally consistent between record statements, both in terms of\nthe data type (e.g. integer, floating-point) and in what it\nrepresents. Otherwise, the graphs will not make sense. The data type\nfor the graphs is taken from the format string for the first `RECORD`\nstatement exporting data to that channel.\n\n\n## Recorder trace value\n\nThe `RECORDER_TRACE(name)` macro lets you test if the recorder trace\nfor `name` is activated. The associated value is an `intptr_t`.\nAny non-zero value activates tracing for that recorder.\n\nYou may set the value for `RECORDER_TRACE` to any value that fits in\nan `intptr_t` with `recorder_trace_set`. This can be used for example\nto activate special, possibly expensive instrumentation.\n\nFor example, to measure the duration and average of a function over N\niterations, you could use code like the following:\n\n    if (RECORDER_TRACE(foo_loops))\n    {\n        intptr_t loops = RECORDER_TRACE(foo_loops);\n        record(foo_loops, \"Testing foo() duration\");\n        uintptr_t start = recorder_tick();\n        double sum = 0.0;\n        for (int i = 0; i \u003c loops; i++)\n            sum += foo();\n        uintptr_t duration = recorder_tick() - start;\n        record(foo_loops, \"Average duration %.3f us, average value %f\",\n               1e6 * duration / RECORDER_HZ / loops,\n               sum / loops);\n    }\n\nWhile this is less-often useful, it is also possible to assign to a\nrecorder trace value, for example:\n\n    RECORDER_TRACE(foo_loops) = 1000;\n\nA given instrumentation or program behavior may require multiple\nconfigurable options, or *tweaks*. The `RECORDER_TWEAK_DEFINE` defines\na tweak adn its initial value. The `RECORDER_TWEAK` macro accesses the\nvalue of a tweak (at a cost comparable to accessing a global\nvariable).\n\n    RECORDER_TWEAK_DEFINE(foo_loop, 600, \"Number of foo iterations\");\n    for (int i = 0; i \u003c RECORDER_TWEAK(foo_loops); i++) { foo(); }\n\n\n# Custom recorder data types\n\nOften, you may have your own custom data types in your program. The\nflight recorder has provision for enabling specific format characters\nto callback your own code for specific rendering. For example, if\nyou have a `person` structure defined as:\n\n    typedef struct person { const char *name; unsigned age; } person_t;\n\nYou can make it so that the 'P' character denotes a person pointer,\nand render it as a 'person'. This would be done by adding a rendering\nfunction of type `recorder_type_fn` for character `P` as follows:\n\n    recorder_configure_type('P', render_person);\n\nwhere the function `render_person` could be defined as follows:\n\n    size_t render_person(intptr_t tracing,\n                         const char *fmt, char *buf, size_t len, uintptr_t data)\n    {\n        person_t *p = (person_t *) data;\n        return tracing\n            ? snprintf(buf, len, \"person(%s,%u)\", p-\u003ename, p-\u003eage)\n            : snprintf(buf, len, \"person(%p)\", p);\n    }\n\nBeware that this will overwrite any regular behaviour 'P' might have\nin a printf format, so be careful to use characters that are either\nunused by regular `printf`, or at least not used by your own program\n(e.g. `%v`, `%E`).\n\nThe `tracing` flag contains the trace value for that recorder.\nLike for strings, it may be safe to print more information during\ntracing than at dump time, as shown in the example above. The lower\nbit of the tracing flag will also be set if the `+` modifier was\npassed in the format. So if you know that you are passing the address\nof a global `person_t` variable, you may use the `\"%+P\"` format.\n\n\n## Reacting to signals\n\nIt is often desirable to dump the recorder when some specific signal is\nreceived. To detect crashes, for example, you can dump the recorder\nwhen receiving `SIGBUS`, `SIGSEGV`, `SIGILL`, `SIGABRT`, etc. To do\nthis, call the function `record_dump_on_signal`\n\n    recorder_dump_on_signal(SIGBUS);\n\nWhen running BSD or macOS, you can have your program dump the current\nstate of the recorder by adding a signal handler for `SIGINFO`. You\ncan then dump the recorder at any time by pressing a key (typically\nControl-T) in the terminal.\n\nIn order to dump the recorder on common signals, the call\n`recorder_dump_on_common_signals (0,0)` will install handlers for the\nfollowing signals if they exist on the platform:\n\n* `SIGQUIT`\n* `SIGILL`\n* `SIGABRT`\n* `SIGBUS`\n* `SIGSEGV`\n* `SIGSYS`\n* `SIGXCPU`\n* `SIGXFSZ`\n* `SIGINFO`\n* `SIGUSR1`\n* `SIGUSR2`\n* `SIGSTKFLT`\n* `SIGPWR`\n\nThe two arguments are bitmask that you can use to add or remove\nsignals. For instance, if you want to get a recorder dump on `SIGINT`\nbut none on `SIGSEGV` or `SIGBUS`, you can use:\n\n    unsigned enable = 1U \u003c\u003c SIGINT;\n    unsigned disable = (1U \u003c\u003c SIGSEGV) | (1U \u003c\u003c SIGBUS);\n    recorder_dump_on_common_signals(enable, disable);\n\n\n## Performance considerations\n\nThe flight recorder `RECORD` statement is designed to cost so little\nthat you should be able to use it practically anywhere, and in\npractically any context, including in signal handlers, interrupt\nhandlers, etc.\n\nAs shown in the data below, a `RECORD` call is faster than a\ncorresponding call to`snprintf`, because text formatting is only done\nat dump time. It is comparable to the cost of a best-case `malloc`\n(directly from the free list), and faster than the cost of a typical\n`malloc` (with random size).\n\nMost of the cost is actually from keeping track of time, i.e. updating\nthe `timestamp` field. If you need to instrument the tightest loops in\nyour code, the `RECORD_FAST` variant can be about twice as fast by\nreusing the last time that was recorded in that recorder.\n\nThe following figures can help you compare `RECORD` to\nvarious low-cost operations. In all cases, the message being recorded\nor printed was the same, `\"Speed test %u\", i`:\n\nFunction                        | Xeon  | Mac   | Pi    | Pi-2  |\n--------------------------------|-------|-------|-------|-------|\n`record_fast`                   |  20ns |  20ns | 129ns |  124ns|\n`record`                        |  35ns |  64ns |1070ns |  726ns|\n`gettimeofday`                  |  16ns |  36ns | 913ns |  675ns|\n`memcpy` (512 bytes)            |  26ns |  15ns |1669ns |  499ns|\n`malloc` (512 bytes)            |  40ns |  61ns | 603ns |  499ns|\n`snprintf`                      |  64ns |  98ns |2530ns | 1071ns|\n`fprintf`                       |  64ns |  14ns |4840ns | 2318ns|\nFlushed `fprintf`               | 751ns |1334ns |4509ns |14730ns|\n`malloc` (512-4K jigsaw)        | 508ns | 483   |3690ns | 5466ns|\nHanoi 20 (printing piped to wc) | 320ms | 180ms |19337ms| 7110ms|\nHanoi 20 (recording)            |  60ms |  60ms | 1203ms|  834ms|\nHanoi 20 (fast recording)       |  23ms |  24ms |  262ms|  175ms|\n\n\nScalability depending on number of threads\n\nFunction                    | Xeon  | Mac   | Pi    | Pi-2  |\n----------------------------|-------|-------|-------|-------|\n`record_fast` * 1           |  20ns | 21ns  | 137ns | 133ns |\n`record_fast` * 2           |  92ns | 86ns  | 137ns | 110ns |\n`record_fast` * 4           |  94ns | 76ns  | 137ns | 152ns |\n`record_fast` * 8           |  57ns | 55ns  | 137ns | 152ns |\n`record_fast` * 16          |  52ns | 52ns  | 137ns | 152ns |\n`record_fast` * 32          |  52ns | 54ns  | 137ns | 152ns |\n`record` * 1                |  37ns | 60ns  |1315ns | 742ns |\n`record` * 2                |  97ns | 93ns  |1076ns | 412ns |\n`record` * 4                |  95ns | 71ns  |1080ns | 224ns |\n`record` * 8                |  59ns | 54ns  |1083ns | 224ns |\n`record` * 16               |  54ns | 50ns  |1084ns | 224ns |\n`record` * 32               |  54ns | 53ns  |1372ns | 224ns |\n\n\nThe platforms that were tested are:\n\n* Xeon: a 6-CPU (12-thread) Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz\n  running Fedora 26 Linux kernel, GCC 7.1.1\n\n* Mac: a 4-CPU (8-thread) 2.5GHz Intel Core i7 MacBook Pro (15'\n  Retina, mid 2015), Xcode 8.1.0 clang\n\n* Pi: First generation Raspberry Pi, ARMv6 CPU, running Raspbian Linux\n  with kernel 4.4.50, GCC 4.9.2\n\n* Pi-2: Second generation Raspberry Pi, 4-way ARMv7 CPU, running\n  Raspbian Linux with kernel 4.4.50, GCC 4.9.2\n\nIt is possible to disable even recording entirely by setting the trace value to\n`-1`, or using the `/` prefix in a trace specification. When recording is\ndisable, the cost of the associated `record` entries has been measured to be\nabout 3ns on the Mac test platform.\n\n\n## Multithreading considerations\n\nThe example of recorder dump given at the beginning of this document\nshows record entries that are printed out of order, and with\nnon-monotonic time stamps.\n\nHere is an example of non-monotonic timestamp (notice that time for\nthread 7 is ahead of time for thread 3 and thread 15):\n\n    recorder_test.c:98: [198750757 9.999929] SpeedTest: [thread 3] Recording 12448096\n    recorder_test.c:98: [198750760 9.999955] SpeedTest: [thread 7] Recording 12527067\n    recorder_test.c:98: [198750758 9.999929] SpeedTest: [thread 15] Recording 12379912\n\nHere is an example of the entries being out of order (notice that the\norder ending in 80 is between those ending in 78 and 79):\n\n    recorder_test.c:98: [198750778 9.990261] SpeedTest: [thread 2] Recording 12374034\n    recorder_test.c:98: [198750780 9.999930] SpeedTest: [thread 8] Recording 12446849\n    recorder_test.c:98: [198750779 9.999930] SpeedTest: [thread 15] Recording 12379916\n\n\nThis is normal behaviour under heavy load, but requires an\nexplanation. The `record` statements can be performed simultaneously\nfrom multiple threads. If there is \"contention\", i.e. if multiple CPUs\nare attempting to write at the same time, one CPU may acquire its\norder and timestamp *before* another CPU, but may end up writing the\nrecord *after* that other CPU. The same thing may also happen if the\noperating system suspends a thread while it is writing the record, in\nwhich case a timestamp discrepancy of several milliseconds may appear\nbetween nearby records.\n\nIn general, this should have a minimal impact on the understanding of\nwhat is happening, and may help you pinpoint risks of race condition\nin your code.\n\n## Recorder scope\n\nThe recorder scope application, `recorder_scope`, shows real-time graphs\nof selected `record` statements in your program.\n\n![Recorder scope](scope/Scope.png)\n\nThe `recorder_scope` application is not built by default.  In order to\nbuild it, you need Qt5 and Qt5 Charts. You can then build it using\n`make scope` from the top-level.\n\nThe `recorder_scope` application uses the recorder API to remotely access\na program being monitored. This communication happens over shared memory,\nusing a file specified in the `RECORDER_SHARE` environment variable.\nThat value needs to be the same for both the program being monitored and\nthe matching `recorder_scope` program. By default, the file name is\n`/tmp/recorder_share`.\n\n### Command-line arguments\n\nThe `recorder_scope` application takes the following command-line arguments:\n\n* A channel-regexp to match exported channels. Each channel-regexp builds\n  a new graph, and graphs show stacked in a single window. If you want\n  multiple windows for the same channel, you can run multiple instances\n  of `recorder_scope`.\n\n* `-c config` sends the given recorder configuration to the program\n  being monitored. It is equivalent to adding the given configuration\n  commands to the `RECORDER_TRACES` environment variable.\n\n* `-s name=init:min:max` creates a slider with a value that can be set\n  between `min` and `max` and starts at `init`. This slider then sends\n  configuration commands to the `name` tweak. This can be used to\n  adjust parameters in the monitored program on the fly.\n\n* `-d delay` sets the number of seconds that can be shown at once on\n  screen.  This is useful for example if the sampling rate for the\n  events is not regular.\n\n* `-w samples` sets the number of samples that can be displayed.  The\n  default is 0, which corresponds to the width of the window. Using a\n  larger value makes it possible to show more data, but at the expense\n  of performance and, in some cases, readability of the data.\n\n* `-t` toggles the display of a time graph for the following graphs. The\n  time graph is an additional graph showing timing information about\n  the event being recorded. It is drawn on a microseconds scale that shows\n  on the right of the graph.\n  ![Timing graph](scope/Timing.png)\n\n* `-m` toggles the display of a min/max values graph for the following\n   graphs\n   ![Min, max and average](scope/MinMaxAverage.png)\n\n* `-a` toggles the display of the average value for the following graphs\n\n* `-n` toggles the display of the normal value for the following graphs,\n  which is useful if you want to remove it from display for clarity.\n\n* `-r ratio` sets the rolling average ratio used for `-m` and `-a` options.\n  It is a percentage, and should be set relatively high. The default is\n  `-r 99`, which means that a new sample changes the average by 1% only.\n\n* `-s basename` sets the basename when saving data. The default is\n  `recorder_scope_data-`. Hitting the `i` key saves an image of the\n  current graph (i.e. the graph with the focus). Hitting the `c` key\n  saves a CSV file with the currently displayed values. Hitting the\n  space bar saves both a file and CSV data. Files are numbered\n  incrementally so as to not overwrite an existing file. When hitting\n  the space bar, the same number will be used for both the image and\n  CSV file.\n\n* `-g WxH` sets the window geometry to WxH pixels.\n\n* `-g WxH@XxY` sets the window geometry to WxH pixels and the window\n  position to X,Y.\n\nIn the recorder scope window, hitting the `t`, `a`, `n` or `m` key\ntoggles the corresponding setting (timing, average, normal and min/max views).\n\n\n### Example\n\nFor example, run the `recorder_test` program with:\n\n    RECORDER_TRACES='SpeedTest=tid,value,mod,delay' ./recorder_test_test 1 1000\n\nTHen run the `recorder_scope` with:\n\n    recorder_scope                      \\\n        -s sleep_time=0:0:100000        \\\n        -s sleep_time_delta=0:0:100000  \\\n        -t mod                          \\\n        -m -a -t delay\n\nThis will open a window that looks like the following:\n\n![Recorder scope with sliders](scope/ScopeAndSliders.png)\n\n\n### Sliders to adjust tweaks\n\nYou can then adjust the waiting time for the program being run\ndynamically using the two sliders. They will update the values of the\ntwo recorder tweaks defined in `recorder_test.c` as follows:\n\n    RECORDER_TWEAK_DEFINE(sleep_time, 0, \"Sleep time between records\");\n    RECORDER_TWEAK_DEFINE(sleep_time_delta, 0, \"Variations in sleep time between records\");\n\nIn that specific case, the tweaks adjust the wait time in the\nfollowing code in `recorder_test.c`:\n\n   if (RECORDER_TWEAK(sleep_time))\n   {\n       struct timespec tm;\n       uint64_t wait_time = (uint64_t)\n           (RECORDER_TWEAK(sleep_time) + drand48()*RECORDER_TWEAK(sleep_time_delta));\n       tm.tv_sec  = 0;\n       tm.tv_nsec = wait_time * 1000;\n       nanosleep(\u0026tm, NULL);\n   }\n\n### Graphing\n\nThe `RECORDER_TRACES` has indicated that you want to share the\n`SpeedTest` record as four columns, named `tid`, `value`, `mod` and\n`delay`. These corresponds to the four arguments in the following\n`record` in the `recorder_test.c` program:\n\n        record(SpeedTest, \"[thread %u] Recording %u, mod %u after\n %ld\",\n            tid, i, i % 500, current_time - last_time);\n\n\n### Derived values (min, max, average, timing)\n\nThe `recorder_scope` will then graph the `mod` and `delay` columns in\ntwo separate graphs. Since we have activated timing before `mod`,\nthe `recorder_scoope` will show timing information in the `mod` graph.\nSimilaryl, since we have activated min/max and average, and then\ntoggled timing off by using `-t` a second time, the `delay` graph will\nshow min, max and average value, but not timing.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fc3d%2Frecorder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fc3d%2Frecorder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fc3d%2Frecorder/lists"}