{"id":18560776,"url":"https://github.com/eepp/yactfr","last_synced_at":"2025-04-10T02:31:11.595Z","repository":{"id":53210962,"uuid":"135329080","full_name":"eepp/yactfr","owner":"eepp","description":"yet another CTF reader: a CTF reading library offering a C++14 API","archived":false,"fork":false,"pushed_at":"2024-05-17T02:42:56.000Z","size":3174,"stargazers_count":4,"open_issues_count":1,"forks_count":5,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-24T15:41:58.665Z","etag":null,"topics":["common-trace-format","cpp14","ctf","decoder","library","tracing"],"latest_commit_sha":null,"homepage":"","language":"Roff","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eepp.png","metadata":{"files":{"readme":"README.adoc","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-05-29T17:14:26.000Z","updated_at":"2024-05-17T02:42:59.000Z","dependencies_parsed_at":"2024-05-07T17:06:10.895Z","dependency_job_id":null,"html_url":"https://github.com/eepp/yactfr","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eepp%2Fyactfr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eepp%2Fyactfr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eepp%2Fyactfr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eepp%2Fyactfr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eepp","download_url":"https://codeload.github.com/eepp/yactfr/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248144186,"owners_count":21054881,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["common-trace-format","cpp14","ctf","decoder","library","tracing"],"created_at":"2024-11-06T22:04:33.813Z","updated_at":"2025-04-10T02:31:08.425Z","avatar_url":"https://github.com/eepp.png","language":"Roff","funding_links":[],"categories":[],"sub_categories":[],"readme":"// Render with Asciidoctor\n\n= yactfr\nPhilippe Proulx\n3 April 2024\n:idprefix:\n:idseparator: -\nifdef::env-github[]\n:toc: macro\nendif::env-github[]\nifndef::env-github[]\n:toc: left\nendif::env-github[]\n\n_**yactfr**_ (yac·tif·er) is _**y**et **a**nother **CTF r**eader_.\n\nThe yactfr library is written in pass:[C++14] and offers a (🥁...)\npass:[C++14] API.\n\nWhile the https://diamon.org/ctf/[CTF] reading libraries that I know\nabout focus on decoding and providing you with complete, ordered event\nrecord objects, the yactfr API offers a lower level of CTF processing:\nyou iterate individual element sequences to obtain _elements_:\nbeginning/end of packet, beginning/end of event record, beggining/end of\nstructure, individual data stream scalar values like fixed-length\nintegers, fixed-length floating point numbers, and null-terminated\nstrings, specific clock value update, known data stream ID, and the\nrest. This means the sizes of event records and packets don't influence\nthe performance or memory usage of yactfr.\n\nSome use cases where this library can prove useful:\n\n* Trace inspection programs, for example when you need to debug a custom\n  CTF producer.\n\n* Foundation of an API with a higher level of abstraction.\n\n* Targeted offline analyses with low memory usage.\n\nThe name _yactfr_ takes its inspiration from\nhttps://lloyd.github.io/yajl/[yajl], a JSON parser with a similar\napproach.\n\nWARNING: yactfr is not mature yet! Its API is still experimental:\nthe interface could change at any time.\n\nifdef::env-github[]\ntoc::[]\nendif::env-github[]\n\n== Notable features\n\n* Full https://diamon.org/ctf/v1.8.3/[CTF{nbsp}1.8.3] and\n  https://diamon.org/ctf[CTF{nbsp}2] support.\n\n* Only depends on http://www.boost.org/[Boost] at _build time_, and\n  nothing at run time (except for your pass:[C++] runtime library):\n+\n----\n$ ldd libyactfr.so\n        linux-vdso.so.1 (0x00007ffe98fa5000)\n        libstdc++.so.6 =\u003e /usr/lib/libstdc++.so.6 (0x00007fb8337d2000)\n        libm.so.6 =\u003e /usr/lib/libm.so.6 (0x00007fb83343d000)\n        libgcc_s.so.1 =\u003e /usr/lib/libgcc_s.so.1 (0x00007fb833225000)\n        libc.so.6 =\u003e /usr/lib/libc.so.6 (0x00007fb832e69000)\n        /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb833e51000)\n----\n\n* Simple, documented pass:[C++14] API with a focus on immutability.\n\n* Efficient data stream decoding: zero memory allocation, zero smart\n  pointer reference count change, zero virtual method call, and zero\n  copy during an element sequence iteration.\n\n** yactfr decodes a data stream fixed-length bit array in place: the\n   result is available as an element member.\n\n** A data stream string/BLOB is available as one or more raw data parts\n   in consecutive elements. The actual raw data is a pointer to the\n   current data block of the data source (for example, from a memory\n   mapped region), which means zero string/BLOB copy.\n+\nThis is possible because, as per CTF{nbsp}1.8.3 and CTF2-SPEC-2.0,\ndata stream strings and BLOBs have an 8-bit alignment requirement.\n\n* Decodes CTF packets of any size and event records of any size with\n  steady memory usage and performance.\n\n* Offers a memory mapped file view data source, but you can also\n  implement your own data source.\n\n* It's safe to use two different iterators on the same element sequence\n  in two different threads.\n\n* Full CTF metadata text inspection and validation: a parsing error\n  exception contains a list of locations (offset in bytes, line number,\n  and column number) with associated context messages.\n\n* Full data stream packet decoding information with the\n  http://en.cppreference.com/w/cpp/concept/InputIterator[input iterator]\n  interface, for example:\n\n** The decoded packet magic number.\n** The decoded metadata stream UUID.\n** The expected total length of the current packet.\n** The expected content length of the current packet.\n** The default clock is updated.\n** The next data stream type to use.\n** The next event record type to use.\n** The data stream (instance) ID.\n\n* \"`Seek packet`\" iterator method to seek the beginning of an element\n  known to be located at a specific offset, in bytes, within the same\n  element sequence.\n+\nYou can use this method with non-CTF packet index information, for\nexample http://lttng.org/[LTTng]'s packet index files (not directly\nsupported by yactfr).\n\n* Decoding error reporting with precise message, offset in element\n  sequence (bits) where this message applies within the element\n  sequence, and other relevant properties.\n\n* Performance (`nop` iteration loop) is similar to\n  https://diamon.org/babeltrace/[Babeltrace]{nbsp}1.5.3's (`-o dummy`)\n  with a release \u003c\u003cbuild,build\u003e\u003e.\n+\nI don't publish numbers: experiment by yourself to confirm or deny this\nclaim.\n\n== Limitations\n\nNote that after a few years of working with all sorts of real life CTF\ntraces, I never caught one which yactfr would refuse to decode, but\nI'm still honest:\n\n* Only builds on a Linux/Unix platform.\n+\nThe non-portable part is the memory mapped file source which uses system\nfunctions such as `open()`, `close()`, `fstat()`, `mmap()`, and\n`madvise()`.\n\n* Decodes up to, and including, 64-bit signed/unsigned CTF fixed-length\n  bit arrays and integers (no \"`big integers`\").\n\n* Decodes up to, and including, 63-bit (effective, which means\n  72{nbsp}total) signed/unsigned CTF variable-length integers.\n\n* Only decodes 32-bit and 64-bit CTF fixed-length floating point\n  numbers.\n\n* Packet lengths must be multiples of 8 bits (I'm still not sure, but\n  this could be enforced by the specification anyway), and be at least\n  8{nbsp}bits.\n\n* TSDL (CTF{nbsp}1.8): Doesn't handle single-line comment continuation\n  when the line ends with `\\`:\n+\n--\n----\nevent {\n    name = \"hello\"; // this is a comment \\\n    id = 23; we're still in the comment started above here\n    id = 42;\n    ...\n};\n----\n--\n\n* TSDL (CTF{nbsp}1.8): Doesn't support relative dynamic-length array\n  type lengths and variant type selectors in data type aliases (or named\n  structure/variant types) which target structure member types outside\n  this data type alias.\n+\nFor example, this is not supported (TSDL):\n+\n--\n----\nfields := struct {\n    int len;\n\n    typealias struct {\n        int sequence[len];\n    } := my_struct;\n\n    struct {\n        int len;\n        my_struct a_struct;\n    } field;\n};\n----\n--\n+\nThis is also not supported (TSDL):\n+\n--\n----\nfields := struct {\n    enum {\n        ...\n    } tag;\n\n    variant my_variant \u003ctag\u003e {\n        ...\n    } a_variant;\n\n    my_variant the_variant;\n};\n----\n--\n+\nThe example above would work, however, if the selector location of the\nvariant type would be absolute:\n+\n--\n----\nfields := struct {\n    enum {\n        ...\n    } tag;\n\n    variant my_variant \u003cevent.fields.tag\u003e {\n        ...\n    } a_variant;\n\n    my_variant the_variant;\n};\n----\n--\n\n* API and ABI backward compatibility is not guaranteed at this point.\n+\nPlease rebuild your project if you change the yactfr version.\n\n[[build]]\n== Build and install yactfr\n\nMake sure you have the build time requirements:\n\n* Linux/Unix platform\n* https://cmake.org/[CMake] ≥ 3.10.0\n* pass:[C++14] compiler\n* http://www.boost.org/[Boost] ≥ 1.58\n* **If you build the API documentation**: http://www.stack.nl/~dimitri/doxygen/[Doxygen]\n\n.Build and install yactfr from source\n----\n$ git clone https://github.com/eepp/yactfr\n$ cd yactfr\n$ mkdir build\n$ cd build\n$ cmake -DCMAKE_BUILD_TYPE=release ..\n$ make\n# make install\n----\n\nYou can specify your favorite C and pass:[C++] compilers with the usual\n`CC` and `CXX` environment variables when you run `cmake`, and\nadditional options with `CFLAGS` and `CXXFLAGS`.\n\nSpecify `-DOPT_BUILD_DOC=YES` to `cmake` to enable the HTML API\ndocumentation build (requires Doxygen). The documentation is available\nin `__BUILD__/doc/api/output/html`, where `__BUILD__` is your build\ndirectory.\n\nSpecify `-DCMAKE_INSTALL_PREFIX=__PREFIX__` to `cmake` to install yactfr\nto the `__PREFIX__` directory instead of the default `/usr/local`\ndirectory.\n\nFor example, this is how I run `cmake` for development:\n\n----\n$ CC=clang CXX=clang++ CXXFLAGS='-Wextra -Wall -pedantic' \\\n  cmake .. -DCMAKE_BUILD_TYPE=debug -DOPT_BUILD_DOC=ON\n----\n\nFor production, you should make a release build:\n\n----\n$ CC=clang CXX=clang++ \\\n  cmake .. -DCMAKE_BUILD_TYPE=release -DOPT_BUILD_DOC=ON\n----\n\n== Run the tests\n\nOnce you have \u003c\u003cbuild,built\u003e\u003e the project in the build directory, you\ncan run the tests. You need Python{nbsp}3 and\nhttps://pytest.org/[pytest].\n\n.Run the yactfr tests from the build directory.\n----\n$ make check\n----\n\nIf you're in a hurry and you have the\nhttps://pypi.org/project/pytest-xdist/[pytest-xdist] package, you can\nparallelize the testing process. You need to set the `YACTFR_BINARY_DIR`\nenvironment variable to the build directory (absolute path), for\nexample:\n\n.Run the yactfr tests in parallel from the build directory.\n----\n$ make tests\n$ YACTFR_BINARY_DIR=$(pwd) pytest -n logical\n----\n\n== Usage examples\n\nIn the examples below, the program accepts two arguments:\n\n. The path to the metadata stream file of the trace (required).\n\n. The path to a data stream file of the same trace (required by some\n  example).\n\n\u003c\u003cbuild,Build\u003e\u003e the API documentation for a thorough reference.\n\nNOTE: The examples are not necessarily optimal: their purpose is to show\nwhat the yactfr API looks like.\n\n.Print all the data stream's event record names.\n====\n[source,cpp]\n----\n#include \u003ccassert\u003e\n#include \u003cfstream\u003e\n#include \u003ciostream\u003e\n#include \u003cyactfr/yactfr.hpp\u003e\n\nint main(const int argc, const char * const argv[])\n{\n    assert(argc == 3);\n\n    // open metadata stream file\n    std::ifstream metadataFile {argv[1], std::ios::binary};\n\n    // create metadata stream object\n    const auto metadataStream = yactfr::createMetadataStream(metadataFile);\n\n    // we have the metadata text at this point: safe to close the file\n    metadataFile.close();\n\n    // get a trace type from the metadata text\n    auto traceTypeMsUuidPair = yactfr::fromMetadataText(metadataStream-\u003etext());\n\n    // create a memory mapped file view factory to read the data stream file\n    yactfr::MemoryMappedFileViewFactory factory {argv[2]};\n\n    // create an element sequence from the trace type and data source factory\n    yactfr::ElementSequence seq {*traceTypeMsUuidPair.first, factory};\n\n    // print all the event record names\n    for (auto\u0026 elem : seq) {\n        if (elem.isEventRecordInfoElement()) {\n            auto\u0026 erInfo = elem.asEventRecordInfoElement();\n\n            // the name of an event record type is optional\n            if (erInfo.type()-\u003ename()) {\n                std::cout \u003c\u003c *erInfo.type()-\u003ename() \u003c\u003c std::endl;\n            }\n        }\n    }\n}\n----\n====\n\n.Print all the fixed-length signed integers of the `sched_switch` event records and their offset.\n====\n[source,cpp]\n----\n#include \u003ccassert\u003e\n#include \u003cfstream\u003e\n#include \u003ciostream\u003e\n#include \u003cyactfr/yactfr.hpp\u003e\n\nint main(const int argc, const char * const argv[])\n{\n    assert(argc == 3);\n\n    // open metadata stream file\n    std::ifstream metadataFile {argv[1], std::ios::binary};\n\n    // create metadata stream object\n    const auto metadataStream = yactfr::createMetadataStream(metadataFile);\n\n    // we have the metadata text at this point: safe to close the file\n    metadataFile.close();\n\n    // get a trace type from the metadata text\n    auto traceTypeMsUuidPair = yactfr::fromMetadataText(metadataStream-\u003etext());\n\n    // create a memory mapped file view factory to read the data stream file\n    yactfr::MemoryMappedFileViewFactory factory {argv[2]};\n\n    // create an element sequence from the trace type and data source factory\n    yactfr::ElementSequence seq {*traceTypeMsUuidPair.first, factory};\n\n    // print all the fixed-length signed integers of the `sched_switch` ERs\n    const auto endIt = seq.end();\n    bool inSchedSwitchEventRecord = false;\n\n    for (auto it = seq.begin(); it != endIt; ++it) {\n        if (it-\u003eisEventRecordInfoElement()) {\n            auto\u0026 ertElem = it-\u003easEventRecordInfoElement();\n\n            // the name of an event record type is optional\n            if (ertElem.type()-\u003ename() \u0026\u0026 *ertElem.type()-\u003ename() == \"sched_switch\") {\n                std::cout \u003c\u003c \"---\" \u003c\u003c std::endl;\n                inSchedSwitchEventRecord = true;\n            } else {\n                inSchedSwitchEventRecord = false;\n            }\n\n            continue;\n        }\n\n        if (inSchedSwitchEventRecord \u0026\u0026 it-\u003eisFixedLengthSignedIntegerElement()) {\n            std::cout \u003c\u003c it.offset() \u003c\u003c \": \";\n\n            auto\u0026 intElem = it-\u003easFixedLengthSignedIntegerElement();\n\n            if (intElem.structureMemberType()) {\n                std::cout \u003c\u003c intElem.structureMemberType()-\u003edisplayName() \u003c\u003c \": \";\n            }\n\n            std::cout \u003c\u003c intElem.value() \u003c\u003c std::endl;\n        }\n    }\n}\n----\n====\n\n.Print all the packet offsets and lengths (both in bits): slow version.\n====\nIn this example, we iterate _all_ the elements of the data stream. The\nnext example shows how to do the same faster.\n\n[source,cpp]\n----\n#include \u003ccassert\u003e\n#include \u003cfstream\u003e\n#include \u003ciostream\u003e\n#include \u003ciomanip\u003e\n#include \u003cyactfr/yactfr.hpp\u003e\n\nint main(const int argc, const char * const argv[])\n{\n    assert(argc == 3);\n\n    // open metadata stream file\n    std::ifstream metadataFile {argv[1], std::ios::binary};\n\n    // create metadata stream object\n    const auto metadataStream = yactfr::createMetadataStream(metadataFile);\n\n    // we have the metadata text at this point: safe to close the file\n    metadataFile.close();\n\n    // get a trace type from the metadata text\n    auto traceTypeMsUuidPair = yactfr::fromMetadataText(metadataStream-\u003etext());\n\n    // create a memory mapped file view factory to read the data stream file\n    yactfr::MemoryMappedFileViewFactory factory {argv[2]};\n\n    // create an element sequence from the trace type and data source factory\n    yactfr::ElementSequence seq {*traceTypeMsUuidPair.first, factory};\n\n    // print all the packet offsets and lengths (both in bits)\n    const auto endIt = seq.end();\n    yactfr::Index curPktOffset = 0;\n    unsigned long curPktNumber = 0;\n\n    for (auto it = seq.begin(); it != endIt; ++it) {\n        if (it-\u003eisPacketBeginningElement()) {\n            // save packet beginning offset\n            curPktOffset = it.offset();\n        } else if (it-\u003eisPacketEndElement()) {\n            // back to first level: end of packet\n            const auto pktLen = it.offset() - curPktOffset;\n\n            std::cout \u003c\u003c \"Packet #\" \u003c\u003c curPktNumber \u003c\u003c \":    \" \u003c\u003c\n                         \"Offset: \" \u003c\u003c std::setw(10) \u003c\u003c curPktOffset \u003c\u003c \"    \" \u003c\u003c\n                         \"Size: \" \u003c\u003c std::setw(10) \u003c\u003c pktLen \u003c\u003c\n                         std::endl;\n            ++curPktNumber;\n        }\n    }\n}\n----\n====\n\n.Print all the packet offsets and lengths (both in bits): fast version.\n====\nThis is a faster version of the previous example.\n\nInstead of decoding the whole packet to find its length, we use the\nexpected packet total length property of the \"`packet info`\" element.\nThis element is available after the decoder reads the packet context.\nThen, we make the iterator seek the next packet directly.\n\nNote that this example doesn't work if the packet context type doesn't\ncontain an expected packet total length fixed-length unsigned integer,\nin which case the data stream _must_ contain a single packet. This could\nbe detected by inspecting the metadata (trace type) and using the size\nof the whole data stream file as the unique packet total length.\n\n[source,cpp]\n----\n#include \u003ccassert\u003e\n#include \u003cfstream\u003e\n#include \u003ciostream\u003e\n#include \u003ciomanip\u003e\n#include \u003cyactfr/yactfr.hpp\u003e\n\nint main(const int argc, const char * const argv[])\n{\n    assert(argc == 3);\n\n    // open metadata stream file\n    std::ifstream metadataFile {argv[1], std::ios::binary};\n\n    // create metadata stream object\n    const auto metadataStream = yactfr::createMetadataStream(metadataFile);\n\n    // we have the metadata text at this point: safe to close the file\n    metadataFile.close();\n\n    // get a trace type from the metadata text\n    auto traceTypeMsUuidPair = yactfr::fromMetadataText(metadataStream-\u003etext());\n\n    // create a memory mapped file view factory to read the data stream file\n    yactfr::MemoryMappedFileViewFactory factory {argv[2]};\n\n    // create an element sequence from the trace type and data source factory\n    yactfr::ElementSequence seq {*traceTypeMsUuidPair.first, factory};\n\n    // print all the packet offsets and lengths (both in bits)\n    const auto endIt = seq.end();\n    auto it = seq.begin();\n    yactfr::Index curPktOffset = 0;\n    unsigned long curPktNumber = 0;\n\n    while (it != endIt) {\n        if (it-\u003eisPacketBeginningElement()) {\n            // save packet beginning offset\n            curPktOffset = it.offset();\n        } else if (it-\u003eisPacketInfoElement()) {\n            // this element contains the expected total length of the current packet\n            auto\u0026 elem = it-\u003easPacketInfoElement();\n\n            assert(elem.expectedTotalLength());\n            std::cout \u003c\u003c \"Packet #\" \u003c\u003c curPktNumber \u003c\u003c \":    \" \u003c\u003c\n                         \"Offset: \" \u003c\u003c std::setw(10) \u003c\u003c curPktOffset \u003c\u003c \"    \" \u003c\u003c\n                         \"Size: \" \u003c\u003c std::setw(10) \u003c\u003c *elem.expectedTotalLength() \u003c\u003c\n                         std::endl;\n            ++curPktNumber;\n\n            /*\n             * Seek the next packet without iterating the intermediate\n             * elements. The expected offset is in bytes, so we need to\n             * divide what we have by 8.\n             */\n            it.seekPacket((curPktOffset + *elem.expectedTotalLength()) / 8);\n            continue;\n        }\n\n        ++it;\n    }\n}\n----\n====\n\n== Contribute and report bugs\n\nPlease contribute with GitHub pull requests and report bugs as GitHub\nissues.\n\n== Community\n\nSee https://eepp.ca/[eepp.ca].\n\nI'm `eepp` on https://libera.chat/[Libera.Chat] and\nhttps://oftc.net/[OFTC].\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feepp%2Fyactfr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feepp%2Fyactfr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feepp%2Fyactfr/lists"}