{"id":20325767,"url":"https://github.com/aboutcode-org/tracecode-toolkit-strace","last_synced_at":"2025-04-11T20:04:23.213Z","repository":{"id":65209722,"uuid":"80970574","full_name":"aboutcode-org/tracecode-toolkit-strace","owner":"aboutcode-org","description":"Trace software components, packages and files between Development/Source and Deployment/Distribution/Binaries codebases - strace build analysis","archived":false,"fork":false,"pushed_at":"2024-08-29T10:05:30.000Z","size":10869,"stargazers_count":25,"open_issues_count":3,"forks_count":7,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-04-11T20:04:13.903Z","etag":null,"topics":["build","build-analysis","build-tracing","strace","tracing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aboutcode-org.png","metadata":{"files":{"readme":"README-build-tracing.rst","changelog":"CHANGELOG.rst","contributing":"CONTRIBUTING.rst","funding":null,"license":null,"code_of_conduct":"CODE_OF_CONDUCT.rst","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.rst","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-02-05T05:16:14.000Z","updated_at":"2024-12-11T20:50:33.000Z","dependencies_parsed_at":"2024-08-29T11:11:38.406Z","dependency_job_id":"3ab7f897-0f8b-4982-b6a7-4af632ea225a","html_url":"https://github.com/aboutcode-org/tracecode-toolkit-strace","commit_stats":{"total_commits":178,"total_committers":14,"mean_commits":"12.714285714285714","dds":0.4101123595505618,"last_synced_commit":"ab03f41a7bf0b3666fec80c796da1c98af8fb4cf"},"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aboutcode-org%2Ftracecode-toolkit-strace","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aboutcode-org%2Ftracecode-toolkit-strace/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aboutcode-org%2Ftracecode-toolkit-strace/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aboutcode-org%2Ftracecode-toolkit-strace/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aboutcode-org","download_url":"https://codeload.github.com/aboutcode-org/tracecode-toolkit-strace/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248473129,"owners_count":21109628,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["build","build-analysis","build-tracing","strace","tracing"],"created_at":"2024-11-14T19:41:33.844Z","updated_at":"2025-04-11T20:04:23.183Z","avatar_url":"https://github.com/aboutcode-org.png","language":"Python","readme":"Tracing a build on Linux\n=========================\n\nTraceCode is a tool to trace the execution of a build, so you can learn which\nfiles are read, compiled, built and ultimately deployed.\n\nThis document provides guidelines on how to collect the trace a of build to\nanalyze later with TraceCode.\n\nThis trace is collected with an open source tool called strace. \nstrace is LGPL-licensed and available at: \n\nhttps://github.com/strace/strace/\n\nYou must use strace v.4.9 or later either from:\n\n- your Linux distribution (if it has strace 4.9 or higher)\n- built from sources from http://sourceforge.net/project/strace/\n  BUILDING INSTRUCTIONS FOR STRACE ARE PROVIDED IN SECTION 2.\n\n\n0. Basic system requirements on the build host\n========================================================\n\nYou need first:\n\n * a Linux-based build host (the machine on which you run the build).\n * sudo or root access on this build host.\n * plenty of disk space, typically 100GB free space or more for some large builds\n\n\nAdditional reference information:\n\n    The set of files created during the trace of a build can be as big or\n    bigger than your codebase. The rule of thumb is that the space needed to\n    store the trace of a build may be roughly as big as the space needed to\n    store the combination of all the intermediate files (such as .o and\n    .class) and final built files (such as .so and .jar) created during the\n    build. Assuming about 100GB of free space just for storing the build\n    traces uncompressed should be enough to store even the largest build\n    traces.\n\n    Many small files are created during the tracing process, all in one\n    directory (up to 500K files) which could be a problem on some file\n    systems. This is usually not a problem with recent filesystems such as\n    ext4 or similar.\n\n\n\n1. Increase the PIDs limit of the build machine\n===================================================\n\nTo do this, run this configuration only once::\n\n    $ sudo su -\n    $ echo 4194303 \u003e /proc/sys/kernel/pid_max\n\nthen exit root with Ctrl-D\n\n\nAdditional reference information:\n\n    For large builds, it may be necessary to increase the pid_max (the max\n    number of process IDs) which defaults to 32768 in most cases.  Why\n    increase the pid_max? Because strace creates one log file per process\n    spawned during the build, using the PID (process id) as the saved trace\n    file name.  A large build may easily spawn more than 100K processes and we\n    need to have a unique filename per pid. If the max number of PID is\n    reached, the PID restarts at one; previously saved trace files for a given\n    PID could be overwritten with the same PID number.  For instance compiling\n    each source file will spawn a new compiler process and a new trace file\n    will be created for each process.  A Linux kernel contains 30 to 40K files\n    and compiling a kernel may spawn 30 to 40K processes and therefore the PID\n    may rollover to one if you use the default pid_max setting.  Very large\n    and long running builds can spawn up to a million different processes.\n\nYou can check the PID max with::\n\n    $ cat /proc/sys/kernel/pid_max\n    32768\n\n32768 is the typical default value on most Linux installations. To\nincrease the PID to 4194303 use the command below. (note: the value\n4194303 is the highest supported value on 32 bit systems and is high\nenough for all known cases)::\n\n    $ sudo su -\n    $ echo 4194303 \u003e /proc/sys/kernel/pid_max\n    then exist root with Ctrl-D\n\n\n\n2. Build and install strace\n=============================\n\nIf you build strace from sources (as opposed to use a distro package for strace 4.9\nor higher) use these additional instructions.\n\nInstall these packages:\n-------------------------------------------\n\nBasic packages installed on the build host (such as build-essential on \ndebian) and libtool, automake and autoconf.\n\n- On Debian or Debian-derivatives such as Ubuntu you might need something\n  like::\n\n    sudo apt-get install build-essential automake autotools-dev autoconf \\\n    libtool\n\n- On RPM-based Linuxes such as Fedora or OpenSuse you might need something\n  like::\n\n    sudo yum groupinstall \"Development Tools\"\n    sudo yum install automake autotools autoconf libtool\n\nInstall the latest strace by building it from sources at:\n-------------------------------------------------------------\n\n    https://github.com/strace/strace/\n\nAdditional reference information:\n\n    DO NOT USE an strace package from an strace version before v4.9 if provided\n    with your build host Linux distribution: it can be buggy at times or be an\n    older version that does not contain the required features set.\n\nBuild and install strace this way::\n\n    wget http://master.dl.sourceforge.net/project/strace/strace/4.15/strace-4.15.tar.xz\n    tar -xzf 4.15.tar.xz\n    cd strace-4.15\n    autoreconf -i\n    ./configure\n    make\n\nNOTE: installing strace globally in your system is optional.\nYou can run it from its build location too.\n\nTo install globally in /usr/local/bin/strace use::\n\n    sudo make install\n\n\n3. Prepare the build you want to trace\n==========================================\n\n3.1 Ensure your build is ready to run and properly configured.\n-------------------------------------------------------------------\n\nWipe clean and/or disable any compilation cache (such as ccache, bref).\nMake clean or distclean or equivalent.\n\n\nAdditional reference information:\n\nYou MUST ensure that the build is fully cleaned first. Run a make clean or\na similar command to ensure that all artifacts of previous build runs are\ndeleted including:\n\n- clearing object caches if you use caching compiler such as ccache.\nUse ccache --clear to clear a ccache cache.\n\n- clearing download caches if you use library repositories fetched\nremotely at build time such as with maven.\n\n- removing all intermediate and final compiled or built object, archives\nor deployed images\n\n\n3.2 Save a tarball of the initial development codebase before the build.\n--------------------------------------------------------------------------------\n\nCreate a tarball of the whole clean codebase BEFORE running the build. This\ncan include a custom toolchain if you do not use the standard installed\ncompiler and toolchain from your build host. This initial snapshot of the\ncodebase should contain all the files (either source or pre-compiled) that are\nused in the build.\n\nUse this tarball in step 5\n\n\n\n4. Trace your build(s)\n======================\n\nRun as root if your build use sudo or setuid commands.\n\nIf you need multiple commands, trace each command, ensuring you use a\nDIFFERENT output directory for each command.\n\nDo a regular release build, NOT a debug build.\n\n\nAdditional reference information:\n\n    If your build requires more than one command, execute this process once\n    for each command. Ensure that you create a NEW DISTINCT trace output\n    directory (a.k.a. {tracing_dir} for each build command that you trace.\n\n    With strace, programs that use the setuid bit do not have effective user\n    ID privileges while being traced. If your build use setuid privileges such\n    as to create filesystems or special files with mknod or if you build use\n    sudo, you will need to execute strace as ROOT. There are possibilities to\n    use the -u option as an extra setting, passing -u \u003cuser\u003e as the user that\n    should be set by strace when doing setuid related operations.\n\n    Your build should NOT BE INTERACTIVE such as asking password for su or\n    sudo (for example if it needs to create special files as root, rather than\n    using something like fakeroot). In cannot either use commands that use\n    setuid. This is a limitation os strace and the Linux Kernel itself. In\n    this case, run your build as root.\n\n    If your build is run by a user with setuid privileges these privileges\n    will be ignored during tracing and you will need need to run the build as\n    sudo or root too.\n\n    If your build is interactive, you need to find a way to run it with\n    arguments or variables such that it can run end to end as one unattended\n    command. You can write a small wrapper script for this purpose.\n\n\n4.1. Collect the build trace with strace\n------------------------------------------------\n\nTrace each of your build commands with strace, replacing the names in braces\n{} with your actual build command and args, output directory and prefix. Use\n$(which strace) if you installed strace globally or use the path to the strace\nbuild directory if you did not install strace.\n\nExecute your build under strace with this command::\n\n    $(which strace) -ff -y -ttt -qq -a1 \\\n    -o {NEW EMPTY tracing_dir}/{trace prefix: project name, build number or version} \\\n    {build command}\n\nFor instance to run a simple make -j2, use this command::\n\n    mkdir ~/mybuild-trace1\n    $(which strace) -ff -y -ttt -qq -a1 -o ~/mybuild-trace1/myprod-v2 make -j2\n\n\nAdditional reference information:\n    For each build command you need to run, create a NEW EMPTY directory\n    __OUTSIDE of your build and codebase directory tree___ to store contain\n    the output of each traced build command. We will refer to this directory\n    as {tracing_dir}.\n\n    Ensure that you create a NEW EMPTY {tracing_dir} directory for each traced\n    build command. Do not mix several commands traces output in the same\n    directory.\n\n    If a build fails and needs to be restarted, create a NEW EMPTY directory\n    when you rerun the build. Ensure you clean the build entirely before\n    reruning it. Traces of partial builds cannot be interpreted correctly.\n\n\n4.2. Wait for the build to complete\n-------------------------------------------------------\n\nA traced build will take from 3 to 10 times longer than a regular build. Using\na multi-threaded build helps (such as using the -j option with make) if your\nbuild supports it.  A rule of thumb for make is to use -j x+2 where x is the\nnumber of cores on the build machine.\n\n\n4.3. Verify that your build completed correctly\n-----------------------------------------------\n\nCheck that your final build artifacts were created correctly by the build and\nthat your build did not have any error.\n\n\n\n5. Collect archives for the built codebase, traces and build outputs.\n==========================================================================\n\nCreate a tarball of the built codebase after running the build including all\nthe deployed codebase directories and the final deployed images or archives.\n\nMake sure to include the 'out' directory or a similar directory that contains\nall intermediate objects if the build artifacts are not created in-place in\nthe development codebase but in a separate 'out-style' directory.\n\nIt is best if and perfectly OK if this archive also contains again a copy of\nthe full development code.\n\nCreate a tarball of the build traces stored in every {tracing_dir}. Ensure\nthat you create one separate tarball for each traced build command output\ndirectory.\n\nCollect the tarball of the development codebase BEFORE the build created in\nstep 3.2.\n\nYou can then use these archives to trace your build with the tracecode tool.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faboutcode-org%2Ftracecode-toolkit-strace","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faboutcode-org%2Ftracecode-toolkit-strace","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faboutcode-org%2Ftracecode-toolkit-strace/lists"}