{"id":35310864,"url":"https://github.com/project705/guardrails","last_synced_at":"2026-04-13T11:02:36.488Z","repository":{"id":274891529,"uuid":"402956162","full_name":"project705/guardrails","owner":"project705","description":"Yet another memory debugger","archived":false,"fork":false,"pushed_at":"2021-09-08T18:39:44.000Z","size":33,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-30T00:27:55.104Z","etag":null,"topics":["asan","gdb","radare2","valgrind"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/project705.png","metadata":{"files":{"readme":"README.md","changelog":"changes.txt","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-04T03:41:06.000Z","updated_at":"2021-09-09T18:32:58.000Z","dependencies_parsed_at":"2025-01-30T00:37:56.460Z","dependency_job_id":null,"html_url":"https://github.com/project705/guardrails","commit_stats":null,"previous_names":["project705/guardrails"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/project705/guardrails","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/project705%2Fguardrails","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/project705%2Fguardrails/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/project705%2Fguardrails/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/project705%2Fguardrails/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/project705","download_url":"https://codeload.github.com/project705/guardrails/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/project705%2Fguardrails/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31749767,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-13T09:16:15.125Z","status":"ssl_error","status_checked_at":"2026-04-13T09:16:05.023Z","response_time":93,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asan","gdb","radare2","valgrind"],"created_at":"2025-12-30T17:43:11.511Z","updated_at":"2026-04-13T11:02:36.483Z","avatar_url":"https://github.com/project705.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Guardrails Memory Debugger\n\n## Background\n\nGuardrails was written years ago to debug a highly parallel distributed OLAP\ndatabase.  The database had a custom runtime that managed many user-mode fibers\nwith makecontext, swapcontext and getcontext.  Most of the\ntest/verification/automation infrastructure assumed clusters consisting of\nmany-core machines.  This resulted in very poor valgrind performance due to\nthread serialization. Older ASAN implementations had difficulty dealing with\nthe swapcontext calls.  Thus guardrails was born.\n\nThese days you should probably start with tools like ASAN, valgrind, radare,\netc.  However the relative simplicity of the guardrails approach allows a lot\nof flexibility in modifying the tool itself for specific debugging tasks. Note\nthat guardrails performance will be suboptimal under very heavy heap churn of\nsmall allocations. In the case of the OLAP database, this was of minimal impact\nbecause the heap was mostly used for the control path (very complex but\nrelatively lightly loaded) and not the data path, which was the performance\ncritical path but largely relied on larger allocations via a custom slab\nallocator.\n\n## Introduction\n\nGuardRails is a MMU-based memory debugging tool.  The objective is to support\nsimilar functionality to valgrind but with much higher performance in highly\nthreaded environments.  GuardRails is a full replacement for the libc memory\nallocator.  Calls such as malloc, memalign, free, realloc, etc. are all handled\nby GuardRails.  This is achieved using the LD_PRELOAD environment variable to\nforce the loader to overlay the GuradRails shared library over libc.\n\nValgrind is a great debugging tool but is far too slow to run massively\nparallel tests, especially on many-core machines. The Valgrind VM alone running\nmemcheck exhibits a [50x performance degradation over\nnative](http://valgrind.org/docs/manual/manual-core.html#manual-core.whatdoes).\nThe problem is further compounded by the fact that Valgrind [serializes\nthreads](http://valgrind.org/docs/manual/manual-core.html#manual-core.pthreads),\nlimiting execution to a single core regardless of machine size.  These two\nfactors together can result in a \u003cb\u003eperformance degradation factor of over\n1,000x\u003c/b\u003e for many-core machines.  Aside from the performance implications,\nValgrind fully virtualizes program execution, resulting in other undesired\nside-effects related to shared memory, signal handling, etc.\n\n## Guardrails tools\n* `grrun.sh` runs executables under guardrails\n* `test` a very simple test program exercising some guardrails functionality (see `./test -h`)\n* `grgdb.py` GDB debugging macros\n* `grdump.py` leak dumping utility\n* `libguardrails.so.0.0` guardrails shared library; provides dumb best-fit\n  logarithmic allocator along with tracking and protection mechanisms\n\n# GuardRails Debugging Features\n\nGuardRails uses the hardware MMU to detect memory errors by modifying the page\nprotection bits as memory is allocated and freed.  Before actually being\nrecycled freed memory is added to a delay list during which time the pages are\nfully read/write protected.  This ensures that any use-after-free causes a\nsegmentation fault.  Heap header protection is achieved by appending a\nprotected guard page adjacent to all memory allocations.  Any corruption of\nthis header will cause a segfault indicating who did the corruption.\n\nCurrent features include:\n\n* Protection of the heap metadata.  This catches the infamous \"__libc_message\" errors from the libc heap allocator.\n* Buffer overrun detection with allocation and free tracking.\n* Context aware leak tracking with full allocator backtrace.  Both size and frequency of leaks are tracked and reported.\n* Double free detection with allocation and free tracking.\n* Use-after-free detection with allocation and free tracking.\n* Memory poisoning and verification.\n* Delayed free lists.\n* Various OOM handling options.\n* High-watermark tracking with dump intervals.\n* Debugging of transient hidden runtime leaks (leaks that are freed via destructors on program exit).\n* Various stats such as allocation histograms, rates, etc.\n\nFuture features:\n\n* Better test infra\n* Improved use-after-free performance.\n* Buffer underrun detection.\n* Thread-local memory\n* Lockless pools\n\n# Building Guardrails\n\nA full build can be done with:\n\n```console\nmake deps\nmake\n```\n\nThis will result in a shared library `libguardrails.so.0.0` build product and a test program `test`.\n\n# Running Under Guardrails\n\nYou'll may need to increase the `max_map_count` proc node to prevent allocation failures with more complex allocation patterns:\n\n`echo 10000000 | sudo tee /proc/sys/vm/max_map_count`\n\nand\n\n`echo 0 | sudo tee /proc/sys/kernel/randomize_va_space`\n\nThen use the wrapper script to run the program:\n\n`grrun.sh \u003cprogram\u003e`\n\nFor example:\n\n`grrun.sh ./test -o`\n\nto run the simple buffer overrun test (expected to segfault).\n\nNo need to recompile the program; guardrails can run against any arbitrary binary.\n\n# GuardRails Debugging Techniques\n\nGuardRails provides several facilities for both detecting and also rapidly\nfixing memory related bugs. In general, \u003cb\u003e\u003cu\u003ewhen such bugs are detected the program\nwill immediately segfault\u003c/u\u003e\u003c/b\u003e.  Debugging can then commence via an attached gdb\nsession or core file (limited functionality at the moment) using the supplied\nGDB macros.\n\nThe following techniques are useful when running under GuardRails.\n\n## Memory Debugging\n\nIn general GuardRails works by causing the process to segfault the moment\ncorruption occurs.  Examining the backtrace from the core will show exactly\nwhat caused the corruption.  Guardrails also adds significant additional\ncontext, including the original allocator and original free'er of the memory in\nquestion.\n\n### GDB Macros\n\nSeveral macros are provided to facilitate GuardRails debugging.  These can be loaded into GDB as follows:\n\n`(gdb) source grgdb.py`\n\nSee \"help \u003cmacro\u003e\" for up-to-date help and examples.\n\nCurrent macros include:\n\n* \u003cb\u003egr-dump-in-flight\u003c/b\u003e: Dump allocator metadata. This is mostly for dumping\n  in-flight allocations to a file for subsequent post-processing with\n  grdump.py.  This is NOT recommended for interactive debugging.\n\n* \u003cb\u003egr-find-header\u003c/b\u003e: Given an arbitrary address, find the GuardRails header\n  without depending on the delay list.  The output of this command can be used\n  as input to gr-print-addr-info.\n\n* \u003cb\u003egr-find-delay-list\u003c/b\u003e: Finds an address on the delayed free list\n  (requires -d option when running GuardRails).  Any address within an\n  allocated range can be supplied here.  This is useful whenever a segfault\n  occurs, as the faulting address is likely to be on this list, indicating a\n  corruption, use-after-free or double free.\n\n* \u003cb\u003egr-heap-meta-corruption\u003c/b\u003e: Try to determine if a faulting address would\n  cause heap corruption. This command indicates if accessing a given address\n  would cause corruption by determining if the access falls on a guard page.\n\n* \u003cb\u003egr-print-addr-info\u003c/b\u003e: Prints information about an address.  This\n  includes the GuardRails header along with the backtraces of the allocator of\n  the memory and the freer of the memory.  In general this information should\n  result in an almost immediate fix.  Note that the address MUST be either the\n  original address returned by the allocator (eg malloc), or the GuardRails\n  header address.  If the address is some other pointer, use gr-find-header to\n  get the associated header address.\n\n* \u003cb\u003egr-print-segv\u003c/b\u003e: Print segfault address.\n\n#### GDB Macros Example Session\n\nWhat follows is an example session debugging a corruption using GuardRails GDB macros.  It assumes GuardRails was run with the the following command line in `grargs.txt`:\n\n`-dvt30 -T30 -p 175`\n\nGDB session workflow (of course all the standard GDB debugging facilities are also available but omitted here for brevity):\n\n```\nProgram terminated with signal SIGSEGV, Segmentation fault.\n#0  0x00007fc6af5b4818 in json_decref (json=0x7fc6aa5f2fd0) at jansson.h:111\n111         if(json \u0026\u0026 json-\u003erefcount != (size_t)-1 \u0026\u0026 --json-\u003erefcount == 0)\n(gdb) source grgdb.py\n\n# The actual address that caused the segfault\n(gdb) gr-print-segv\n$1 = (void *) 0x7fc6aa5f2fd8\n\n# Find the GuardRails header associated with that address\n(gdb) gr-find-header 0x7fc6aa5f2fd8\nFound valid header at: 0x7fc6aa5f2000\n\n# Dump the header and traces associated with that address\n(gdb) gr-print-addr-info 0x7fc6aa5f2000\nAddress 0x7fc6aa5f2000 is a header\nAddress 0x7fc6aa5f2000 is free\nHeader:\n{magic = 14800349170826394398, binNum = 12, slotNum = 0, usrData = 0x7fc6aa5f2fc0, usrDataSize = 64, next = 0x0, prev = 0x0, allocBt = 0x7fc6aa5f2038}\n\n# Backtrace that allocated the memory:\n================ Allocation Trace: ================\nLine 325 of \"GuardRails.c\" starts at address 0x7fc6af7c0f35 \u003cmemalignInt+357\u003e\nLine 299 of \"../src/lib/libutil/MemTrack.cpp\" starts at address 0xa9363f \u003c_memAlloc(unsigned long, char const*, char const*)+127\u003e\nLine 996 of \"../src/lib/libutil/MemTrack.cpp\" starts at address 0xa960d9 \u003cmemAllocJson(unsigned long)+25\u003e\nLine 341 of \"value.c\" starts at address 0x7fc6af5b91fb \u003cjson_array+11\u003e\nLine 770 of \"load.c\" starts at address 0x7fc6af5b6185 \u003cparse_value+741\u003e\nLine 734 of \"load.c\" starts at address 0x7fc6af5b609a \u003cparse_value+506\u003e\nLine 734 of \"load.c\" starts at address 0x7fc6af5b609a \u003cparse_value+506\u003e\nLine 779 of \"load.c\" starts at address 0x7fc6af5b61be \u003cparse_value+798\u003e\nLine 898 of \"load.c\" starts at address 0x7fc6af5b6426 \u003cparse_json+70\u003e\nLine 959 of \"load.c\" starts at address 0x7fc6af5b65c3 \u003cjson_loads+227\u003e\nLine 499 of \"../src/lib/libqueryparser/QueryParser.cpp\" starts at address 0x7582b6 \u003cQueryParser::jsonParse(char const*, Dag*)+102\u003e\nLine 654 of \"../src/lib/libqueryparser/QueryParser.cpp\" starts at address 0x758d00 \u003cQueryParser::parse(char const*, Dag**, unsigned long*)+240\u003e\nLine 2794 of \"../src/lib/libdag/Retina.cpp\" starts at address 0x7f2f06 \u003cDagLib::importRetina(ApiImportRetinaInput*, ApiOutput**, unsigned long*, bool)+1110\u003e\nLine 37 of \"../src/lib/libapis/ApiHandlerImportRetina.cpp\" starts at address 0x7d1a3c \u003cApiHandlerImportRetina::run(ApiOutput**, unsigned long*)+140\u003e\nLine 73 of \"../src/lib/libapis/ApisRecvObject.cpp\" starts at address 0x79630f \u003cApisRecvObject::run()+639\u003e\nLine 1113 of \"../src/lib/libapis/ApisRecv.cpp\" starts at address 0x795e4b \u003cdoWorkForApiImmediate(void*)+363\u003e\nLine 65 of \"../src/lib/libruntime/DedicatedThread.cpp\" starts at address 0x9b55d8 \u003cDedicatedThread::threadEntryPoint()+120\u003e\nLine 55 of \"../src/lib/libruntime/Thread.cpp\" starts at address 0x9bc911 \u003cThread::threadEntryPointWrapper(void*)+113\u003e\nLine 312 of \"pthread_create.c\" starts at address 0x7fc6ae722173 \u003cstart_thread+179\u003e\nLine 113 of \"../sysdeps/unix/sysv/linux/x86_64/clone.S\" starts at address 0x7fc6acbdc03d \u003cclone+109\u003e\n\n# Backtrace that freed the memory:\n================ Free Trace: ================\nLine 755 of \"../src/lib/libutil/MemTrack.cpp\" starts at address 0xa93e99 \u003cmemFree(void*)+313\u003e\nLine 1002 of \"../src/lib/libutil/MemTrack.cpp\" starts at address 0xa96139 \u003cmemFreeJson(void*)+25\u003e\nLine 240 of \"../src/lib/libqueryparser/QpMap.cpp\" starts at address 0x765c80 \u003cQpMap::parseJson(json_t*, json_error_t*, WorkItem**)+992\u003e\nLine 567 of \"../src/lib/libqueryparser/QueryParser.cpp\" starts at address 0x758718 \u003cQueryParser::jsonParse(char const*, Dag*)+1224\u003e\nLine 654 of \"../src/lib/libqueryparser/QueryParser.cpp\" starts at address 0x758d00 \u003cQueryParser::parse(char const*, Dag**, unsigned long*)+240\u003e\nLine 2794 of \"../src/lib/libdag/Retina.cpp\" starts at address 0x7f2f06 \u003cDagLib::importRetina(ApiImportRetinaInput*, ApiOutput**, unsigned long*, bool)+1110\u003e\nLine 37 of \"../src/lib/libapis/ApiHandlerImportRetina.cpp\" starts at address 0x7d1a3c \u003cApiHandlerImportRetina::run(ApiOutput**, unsigned long*)+140\u003e\nLine 73 of \"../src/lib/libapis/ApisRecvObject.cpp\" starts at address 0x79630f \u003cApisRecvObject::run()+639\u003e\nLine 1113 of \"../src/lib/libapis/ApisRecv.cpp\" starts at address 0x795e4b \u003cdoWorkForApiImmediate(void*)+363\u003e\nLine 65 of \"../src/lib/libruntime/DedicatedThread.cpp\" starts at address 0x9b55d8 \u003cDedicatedThread::threadEntryPoint()+120\u003e\nLine 55 of \"../src/lib/libruntime/Thread.cpp\" starts at address 0x9bc911 \u003cThread::threadEntryPointWrapper(void*)+113\u003e\nLine 312 of \"pthread_create.c\" starts at address 0x7fc6ae722173 \u003cstart_thread+179\u003e\nLine 113 of \"../sysdeps/unix/sysv/linux/x86_64/clone.S\" starts at address 0x7fc6acbdc03d \u003cclone+109\u003e\n\n# Backtrace of the code that tried to use the memory after it was freed:\n(gdb) bt\n#0  0x00007fc6af5b4818 in json_decref (json=0x7fc6aa5f2fd0) at jansson.h:111\n#1  hashtable_do_clear (hashtable=hashtable@entry=0x7fc6aa604fc0) at hashtable.c:148\n#2  0x00007fc6af5b48d9 in hashtable_close (hashtable=0x7fc6aa604fc0) at hashtable.c:215\n#3  0x00007fc6af5b97a1 in json_delete_object (object=0x7fc6aa604fb0) at value.c:77\n#4  json_delete (json=0x7fc6aa604fb0) at value.c:936\n#5  0x00007fc6af5b4834 in json_decref (json=\u003coptimized out\u003e) at jansson.h:112\n#6  hashtable_do_clear (hashtable=hashtable@entry=0x7fc6aa60efc0) at hashtable.c:148\n#7  0x00007fc6af5b48d9 in hashtable_close (hashtable=0x7fc6aa60efc0) at hashtable.c:215\n#8  0x00007fc6af5b97a1 in json_delete_object (object=0x7fc6aa60efb0) at value.c:77\n#9  json_delete (json=0x7fc6aa60efb0) at value.c:936\n#10 0x00007fc6af5b9815 in json_decref (json=\u003coptimized out\u003e) at jansson.h:112\n#11 json_delete_array (array=\u003coptimized out\u003e) at value.c:364\n#12 json_delete (json=0x7fc6aa54efd0) at value.c:939\n#13 0x0000000000758b5d in json_decref (json=0x7fc6aa54efd0) at /usr/include/jansson.h:112\n#14 QueryParser::jsonParse (this=0x7fc66732b790, query=0x7fc67696ea10 \"[{\\\"op\"..., queryGraph=0x7fc491655a80) at ../src/lib/libqueryparser/QueryParser.cpp:619\n#15 0x0000000000758d11 in QueryParser::parse (this=0x7fc66732b790, query=0x7fc67696ea10 \"[{\\\"op\"..., queryGraphOut=0x7fc476ca95b8, numQueryGraphNodesOut=0x7fc476ca95b0) at ../src/lib/libqueryparser/QueryParser.cpp:654\n#16 0x00000000007f2f26 in DagLib::importRetina (this=0x7fc66a42af60, importRetinaInput=0x7fc6a9a28328, importRetinaOutputOut=0x7fc491651030, importRetinaOutputSizeOut=0x7fc491651038, persist=true) at ../src/lib/libdag/Retina.cpp:2794\n#17 0x00000000007d1a44 in ApiHandlerImportRetina::run (this=0x7fc6a9a1ef80, output=0x7fc491651030, outputSize=0x7fc491651038) at ../src/lib/libapis/ApiHandlerImportRetina.cpp:37\n#18 0x0000000000796345 in ApisRecvObject::run (this=0x7fc491651000) at ../src/lib/libapis/ApisRecvObject.cpp:73\n#19 0x0000000000795e4b in apisRecvObjectImmediateHandler (apisRecvObject=0x7fc491651000) at ../src/lib/libapis/ApisRecv.cpp:1112\n#20 doWorkForApiImmediate (args=0x4) at ../src/lib/libapis/ApisRecv.cpp:1138\n#21 0x00000000009b55ec in DedicatedThread::threadEntryPoint (this=0x7fc67382edd8) at ../src/lib/libruntime/DedicatedThread.cpp:65\n#22 0x00000000009bc91e in Thread::threadEntryPointWrapper (arg=0x7fc67382edd8) at ../src/lib/libruntime/Thread.cpp:55\n#23 0x00007fc6ae722184 in start_thread (arg=0x7fc476cb0700) at pthread_create.c:312\n#24 0x00007fc6acbdc03d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111\n```\n\n## SIGUSR2\n\nIssuing SIGUSR2 to \u0026lt;program_name\u0026gt; running under GuardRails:\n\n`pkill -SIGUSR2 \u003cprogram_name\u003e`\n\nwill cause the current GuardRails state to be dumped to stdout and also cause\nGuardRails to write out the grtrack-\u0026lt;tid\u0026gt;.txt leak tracking file\n(assuming the -t and/or -T option was passed).  Included in the state is the\narguments from grargs.txt, which can be used to verify that GuardRails is\nrunning as intended.  If the -v option was passed, a per-memory-bin stats will\nalso be dumped.\n\nNote that depending on the allocator state, it can take several minutes to dump\nthe tracker file.  To determine when it's done, you can do something like\n\n`watch -n1 ls -al grtracker-*.txt`\n\nto see when it stops growing.  lsof can also be used.  Further, the file is\nalso done dumping when the allocator state summary is finished dumping to\nnode.out (can tail -f this).\n\nNote that grdump.py can also handle partial dumps, which might be sufficient\ndepending on the circumstances.\n\n## Leak Tracking\n\nLeak tracking must be enabled by adding the \"-t \u003cframes\u003e\" and/or \"-T \u003cframes\u003e\"\noption to the grargs.txt file.  There are some deep stacks in the protobuf code\npaths.  In general a max frame count of 30 is sufficient to get enough context\nin all such cases.\n\nWhen tracking leaks, GuardRails will dump an output CSV file called\n\"grtrack.txt\" to the current working directory.  This file is dumped on exit\nand also on SIGUSR2.  It can be post-processed by grdump.py.  This script\ngroups and counts all the leak contexts, generates a symbolic backtrace for\neach context, and sorts them by either the total number of bytes leaked or by\nthe frequency of the leak.\n\nIf instead SIGUSR2 is used to dump grtrack.txt, it will contain not only leaked\nmemory but also any in-use memory at the time SIGUSR2 was issued.\n\n### Leak Tracking Examples\n\nTo show the top 100 leaks by frequency:\n\n\u003cpre\u003e\n./grdump.py -f grtrack-\u0026lt;tid\u0026gt;.txt -b \u0026lt;program_name\u0026gt; -c -t 100 |less\nLeaked total of 4,222,380 bytes across 62,116 leaks from 1,639 contexts\n================================ Context      0 / 1,639 ================================\nLeaked 171,008 bytes across 5,344 allocations of 32 bytes each:\n#0  0x00007f457e430de9: ?? ??:0\n#1  0x000000000091fe24: string_create at value.c:655\n#2  0x000000000091ce60: parse_value at load.c:830\n#3  0x000000000091cf6a: parse_object at load.c:734\n#4  0x000000000091cf6a: parse_object at load.c:734\n#5  0x000000000091d08e: parse_array at load.c:779\n#6  0x000000000091d2f6: parse_json at load.c:898\n#7  0x000000000091d46b: json_loads at load.c:959\n#8  0x000000000069996f: QueryParser::jsonParse at QueryParser.cpp:499\n#9  0x0000000000699de8: QueryParser::parse at QueryParser.cpp:648\n#10 0x00000000006f12cb: DagLib::copyRetinaToNewDag at Retina.cpp:5947\n#11 0x00000000006e02ea: Dag::copyDagToNewDag at Dag.cpp:2105\n#12 0x00000000006e16f3: Dag::cloneDagLocal at Dag.cpp:2800\n#13 0x00000000006e291a: Dag::cloneDagGlobalInt [256], unsigned long, char  [256], Dag::CloneFlags) at Dag.cpp:3676\n#14 0x00000000006e2551: Dag::cloneDagGlobal [256], unsigned int, char  [256], Dag::CloneFlags) at Dag.cpp:3488\n#15 0x00000000006ec54b: DagLib::makeRetina at Retina.cpp:2980\n#16 0x00000000006d453b: ApiHandlerMakeRetina::run at ApiHandlerMakeRetina.cpp:50\n#17 0x000000000075b569: processWorkItem at OperatorsFuncTest.cpp:413\n#18 0x000000000076128e: workerRetina at OperatorsFuncTest.cpp:4359\n#19 0x000000000075e6b2: workerMain at OperatorsFuncTest.cpp:4898\n#20 0x00000000007bd835: DedicatedThread::threadEntryPoint at DedicatedThread.cpp:65\n#21 0x00000000007bfe0f: Thread::threadEntryPointWrapper at ??:?\n#22 0x00007f457d598e25: ?? ??:0\n#23 0x00007f457bc5f34d: ?? ??:0\n================================ Context      1 / 1,639 ================================\nLeaked 84,096 bytes across 2,628 allocations of 32 bytes each:\n\nremaining top 99 leaks elided...\n\u003c/pre\u003e\n\nTo show the top 100 by total leaked memory:\n\n\u003cpre\u003e\n./grdump.py -f grtrack-\u0026lt;tid\u0026gt;.txt -b \u0026lt;program_name\u0026gt; -t 100 |less\nLeaked total of 4,222,380 bytes across 62,116 leaks from 1,639 contexts\n================================ Context      0 / 1,639 ================================\nLeaked 902,472 bytes across 1,213 allocations of 744 bytes each:\n#0  0x00007f457e430de9: ?? ??:0\n#1  0x00007f457e18524d: ?? ??:0\n#2  0x000000000078b7c7: resultSetMakeLocal at LocalResultSet.cpp:147\n#3  0x000000000078b401: usrNodeMsg2pcMakeResultSet at UsrNode.cpp:369\n#4  0x00000000006910fa: MsgMgr::doTwoPcWorkForMsg at Message.cpp:3387\n#5  0x00000000007bfcf5: SchedObject::getSchedulable at SchedObject.h:49\n#6  0x00000000007bd551: memBarrier at System.h:65\n#7  0x00007f457bbadd40: ?? ??:0\n================================ Context      1 / 1,639 ================================\nLeaked 171,008 bytes across 5,344 allocations of 32 bytes each:\n\nremaining top 99 leaks elided...\n\u003c/pre\u003e\n\n### Command Line Options\n\nBecause GuardRails is loaded before the loader parses the command line options\na configuration file must be used.  This file contains a single line of options\nin standard getopt format.  Currently the following options are supported:\n\n* ```-a```: Abort on rlimit-induced OOM rather than return NULL.\n* ```-A```: Abort on actual OOM rather than return NULL.\n* ```-D \u003chwm_bytes\u003e```: Dump state every time memory high water mark increases\n  by this interval or greater.\n* ```-d```: Use delayed free list.  This provides very robust user-after-free\n  detection/tracking but currently slows things down by a factor of about 8x.\n  I have a fix planned to make this feature far more performant.\n* ```-m \u003cpct_mem\u003e```: Limit allocation to pct_mem of total system memory.\n* ```-M \u003cmax_request_bytes\u003e```: Abort if a request exceeds max_request_bytes.\n* ```-p \u003cpoison_byte\u003e```: Prepoison allocated memory with poison_byte.\n* ```-s \u003cnum\u003e```: Number of slots to use.  This controls the level of\n  parallelism in GuardRails.\n* ```-t \u003cframes\u003e```: Enable leak detection and \u003cb\u003eallocation\u003c/b\u003e tracking.  The\n  argument specifies the maximum number of stack frames to track (30 seems to\n  capture sufficient context in all paths).  Has some impact on performance but\n  works with the operators test.  The memory overhead for this feature is\n  modest because it usually uses memory that would otherwise be wasted by the\n  allocator.\n* ```-T \u003cframes\u003e```: Enable leak detection and \u003cb\u003efree\u003c/b\u003e tracking.  Works\n  like the -t option but for frees.\n* ```-v```: Verbose mode.  Dumps size-based stats on exit or SIGUSR2.\n\nThese commands must go in a file named `grargs.txt` in the process's working\ndirectory.\n\n`sudo pwdx $(pgrep \u003cprogram_name\u003e)`\n\nNote that GuardRails will display its argument string to stdout upon exit or\nupon SIGUSR2.  For example:\n\n```\n ================ BEGIN GUARDRAILS OUTPUT ================\n Ran with args: -v -t 30 -T 30 -p 175\n Number mem pools used: 2\n ...\n```\n\nThis can be used to verify that the grargs.txt is being parsed as expected.\n\n# Performance\n\nPerformance was measured for two workloads: one with lots of small heap allocations and another where control-path allocations are heap based and data-path allocations are slab-based.\n\n| Performance Degredation Factor vs Native (slab-intensive workload) |       |\n|--------------------------------------------------------------------|-------|\n| Heap Corruption Tracking                                           | 1.00x |\n| Memory Tracking                                                    | 1.10x |\n| Protected Delay List                                               | 1.13x |\n\n| Performance Degredation Factor vs Native (heap-intensive workload) |       |\n|--------------------------------------------------------------------|-------|\n| Heap Corruption Tracking                                           | 1.25x |\n| Memory Tracking                                                    | 4.82x |\n| Protected Delay List                                               | 8.21x |\n\n# Troubleshooting\n\n## Machine hangs, program runs very slow, never completes, etc\n\nThis is probably due to a high-rate leak of small allocations.  Because of\nallocator overheads, GuardRails is inefficient for small allocations.  When\nmany such allocations leak, it can cause the machine to start swapping\nextensively and bring things to a halt.  Swap activity can be viewed in vmstat\nas follows:\n\n vmstat -w 1\n\npay attention to the si (swap in) and so (swap out) columns.  They should be at\nor near zero.  If not there is non-trivial swapping.\n\nTo track down the culprit, issue the following command:\n\n pkill -SIGUSR2 \u0026lt;program_name\u0026gt;\n\nwhich will cause GuardRails to dump the current tracker state to the\ngrtrack-*.txt file.  Although this includes legitimate in-flight allocations,\nit will usually be very obvious where the leak is as you'll see millions of\ntiny allocations at the very top of the leak count list.  This is probably the\nproblem.\n\n## Missing Symbols in Leak Backtraces\n\nBack in the day we did a lot of static linking for various reasons, and the\nprogram loader seemed to use consistent offsets.  This is no longer the case.\nI did a quick update to the leak dumper to pull in the program text base from\n/proc/\u0026lt;pid\u0026gt;/maps but to get all the symbols lookups working we need to\npull in the base addresses of all the shared libraries from maps.  Should be\npretty easy to do based on the way the program text base address is pulled in.\nThen one can just blindly iterate over these bases during symbol lookup and\nstop when a lookup succeeds. Not perfect but should be good enough...\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproject705%2Fguardrails","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fproject705%2Fguardrails","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproject705%2Fguardrails/lists"}