{"id":34748868,"url":"https://github.com/pjsip/cirunner","last_synced_at":"2026-05-16T16:38:14.759Z","repository":{"id":275521423,"uuid":"926245290","full_name":"pjsip/cirunner","owner":"pjsip","description":"Setup, capture and display crash in GitHub action. Works on Linux, Windows, and MacOS","archived":false,"fork":false,"pushed_at":"2025-09-25T07:19:47.000Z","size":756,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-09-25T08:39:56.426Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pjsip.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-02-02T21:59:33.000Z","updated_at":"2025-02-12T04:54:25.000Z","dependencies_parsed_at":"2025-09-25T08:30:26.387Z","dependency_job_id":"8a77aa8f-80f7-4c1e-83a4-d15935338237","html_url":"https://github.com/pjsip/cirunner","commit_stats":null,"previous_names":["pjsip/ci-runner","pjsip/cirunner"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/pjsip/cirunner","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pjsip%2Fcirunner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pjsip%2Fcirunner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pjsip%2Fcirunner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pjsip%2Fcirunner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pjsip","download_url":"https://codeload.github.com/pjsip/cirunner/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pjsip%2Fcirunner/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33109792,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-16T04:41:52.686Z","status":"ssl_error","status_checked_at":"2026-05-16T04:41:52.009Z","response_time":115,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-12-25T04:57:36.723Z","updated_at":"2026-05-16T16:38:14.753Z","avatar_url":"https://github.com/pjsip.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# cirunner\n\n**cirunner** is a Python script to setup, capture, display crash in GitHub action, and\nuploads core dump etc as action artifacts, for Linux, Windows, and MacOS.\n\n## Usage\n\nSample workflow in your repository:\n\n```\nenv:\n  CI_RUNNER: python ${{ github.workspace }}/cirunner/cirunner.py -t 3600 -o ${{ github.workspace }}/artifacts --\njobs:\n  cirunner-demo:\n    runs-on: ubuntu-latest\n    steps:\n    - uses: actions/checkout@v2\n    - name: get and install cirunner\n      run: |\n        git clone --depth 1 https://github.com/pjsip/cirunner.git\n        cirunner/installlinux.sh\n    - name: build your app\n      run: |\n        gcc -o testapp -O0 -g testapp.c\n    - name: run your app via cirunner\n      run: $CI_RUNNER ./testapp\n    - name: upload artifacts on failure\n      if: ${{ failure() }}\n      uses: actions/upload-artifact@v4\n      with:\n        name: ${{ runner.os }}-${{ runner.arch }}-${{ github.job }}-${{ github.run_id }}\n        path: artifacts\n```\n\n(replace `installlinux.sh` with `installmac.sh` and `installwin.ps1` for Mac and Windows).\n\n\nA program is run via the script as follows (for all platforms):\n\n```bash\npython cirunner.py -t 600 -o artifacts -- target_exe arg1 arg2 ..\n```\n\nThe above does the following:\n\n1. it sets up crash handler on the system\n1. it forwards any stdout/stderr output from the program without buffering.\n1. it terminates the program and generate crash dump if the program runs for more than 600 seconds\n   (default is six hours if timeout is not specified)\n1. on crashes, either because of program crash or terminated due to timeout, it does the following:\n\n   - locate the crash dump file\n   - analyze the crash dump with a debugger to show crash location and backtrace of all threads\n   - copy the program, the crash dump file, and other required files (such as `.pdb` file\n     on Windows) to `artifacts` directory (as specified by `-o` option) and upload them as GitHub\n     action artifact, which can be downloaded for offline analysis.\n\n1. it returns using the return value of the program.\n\n\n## Samples\n\nFor examples, see the GitHub action results and yml workflows for each platform in this repository.\n\nAlso see https://github.com/pjsip/pjproject/pull/4288 for the usage on pjproject repository, including\nhow to debug the crash dump file locally.\n\n## Windows Notes\n\n**Installation**\n\nThe installation is by calling `installwindows.ps1` which does the following tasks:\n\n1. Installing registry in `localdumps.reg`\n2. Download [procdump](https://learn.microsoft.com/en-us/sysinternals/downloads/procdump)\n3. Make sure **cdb** (Microsoft console debugger) exists (it does on GitHub Windows runners)\n\nIf you're testing on your local Windows machine, you will need to install **cdb** by following these instructions:\n\n1. Run Windows SDK 10 installer if you haven't installed it, or run from \n   **Add/Remove Program \u003e Windows Software Development Kit -\u003e Modify**.\n2. Select component: **Debugging Tools for Windows**\n\n**Handling crashes**\n\nOnce the registry settings in `localdumps.reg` is installed, any crashes will be stored as minidump\nin `%localprofile%\\Dumps` folder (e.g. `C:\\Users\\pjsip\\Dumps`). **cirunner** then runs **cdb** to analyze\nthe crash. See sample output below.\n\n**Handling timeout**\n\nOn timeout, **cirunner** executes [procdump](https://learn.microsoft.com/en-us/sysinternals/downloads/procdump)\nto create minidump of the program's state, then runs **cdb** to analyze the minidump.\n\n**Sample crash dump output**\n\nThe output of **cdb** is quite extensive, and useful. It shows crash location, stack trace, and stack trace\nof all threads. Click **Show output** below for sample output with bug induced `pjlib-test`.\n\n\u003cdetails\u003e\n  \u003csummary\u003eShow output\u003c/summary\u003e\n\n```\nMicrosoft (R) Windows Debugger Version 10.0.19041.685 AMD64\nCopyright (c) Microsoft Corporation. All rights reserved.\n\n\nLoading Dump File [C:\\Users\\bennylp\\Dumps\\pjlib-test-i386-Win32-vc14-Debug.exe.5644.dmp]\nUser Mini Dump File: Only registers, stack and portions of memory are available\n\nSymbol search path is: srv*\nExecutable search path is:\nWindows 10 Version 19045 MP (4 procs) Free x86 compatible\nProduct: WinNt, suite: SingleUserTS\n19041.1.amd64fre.vb_release.191206-1406\nMachine Name:\nDebug session time: Tue Feb  4 09:51:15.000 2025 (UTC + 7:00)\nSystem Uptime: not available\nProcess Uptime: 0 days 0:00:06.000\n........................\nThis dump file has an exception of interest stored in it.\nThe stored exception information can be accessed via .ecxr.\n(160c.2e90): Access violation - code c0000005 (first/second chance not available)\nFor analysis of this file, run !analyze -v\neax=00000000 ebx=00000000 ecx=00000f9c edx=00e23600 esi=00000003 edi=00000003\neip=770b358c esp=00afdc58 ebp=00afdde8 iopl=0         nv up ei pl nz na po nc\ncs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202\nntdll!NtWaitForMultipleObjects+0xc:\n770b358c c21400          ret     14h\n0:000\u003e cdb: Reading initial command '!analyze -v; ~* k; q'\n*******************************************************************************\n*                                                                             *\n*                        Exception Analysis                                   *\n*                                                                             *\n*******************************************************************************\n\n\nKEY_VALUES_STRING: 1\n\n    Key  : AV.Fault\n    Value: Write\n\n    Key  : Analysis.CPU.Sec\n    Value: 1\n\n    Key  : Analysis.DebugAnalysisProvider.CPP\n    Value: Create: 8007007e on DESKTOP-8I1RHUB\n\n    Key  : Analysis.DebugData\n    Value: CreateObject\n\n    Key  : Analysis.DebugModel\n    Value: CreateObject\n\n    Key  : Analysis.Elapsed.Sec\n    Value: 1\n\n    Key  : Analysis.Memory.CommitPeak.Mb\n    Value: 87\n\n    Key  : Analysis.System\n    Value: CreateObject\n\n    Key  : Timeline.Process.Start.DeltaSec\n    Value: 6\n\n\nNTGLOBALFLAG:  0\n\nAPPLICATION_VERIFIER_FLAGS:  0\n\nCONTEXT:  (.ecxr)\neax=00000064 ebx=009ff000 ecx=00000f9c edx=00e23600 esi=00000f9c edi=001e114a\neip=001ffbc8 esp=00afe590 ebp=00afe59c iopl=0         nv up ei pl nz ac pe nc\ncs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010216\npjlib_test_i386_Win32_vc14_Debug!capacity_test+0x78:\n001ffbc8 c70100000000    mov     dword ptr [ecx],0    ds:002b:00000f9c=????????\nResetting default scope\n\nEXCEPTION_RECORD:  (.exr -1)\nExceptionAddress: 001ffbc8 (pjlib_test_i386_Win32_vc14_Debug!capacity_test+0x00000078)\n   ExceptionCode: c0000005 (Access violation)\n  ExceptionFlags: 00000000\nNumberParameters: 2\n   Parameter[0]: 00000001\n   Parameter[1]: 00000f9c\nAttempt to write to address 00000f9c\n\nPROCESS_NAME:  pjlib-test-i386-Win32-vc14-Debug.exe\n\nWRITE_ADDRESS:  00000f9c\n\nERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.\n\nEXCEPTION_CODE_STR:  c0000005\n\nEXCEPTION_PARAMETER1:  00000001\n\nEXCEPTION_PARAMETER2:  00000f9c\n\nSTACK_TEXT:\n00afe59c 001ffa8b 00afe604 00afe604 00afe604 pjlib_test_i386_Win32_vc14_Debug!capacity_test+0x78\n00afe5ac 002454a3 001e4a16 0022ecab 332a1299 pjlib_test_i386_Win32_vc14_Debug!pool_test+0xb\n00afe604 0024565e 00afe724 00000000 00afe9c4 pjlib_test_i386_Win32_vc14_Debug!run_test_case+0xb3\n00afe61c 00244b58 00afe724 00afe750 00aff988 pjlib_test_i386_Win32_vc14_Debug!basic_runner_main+0x3e\n00afe62c 002150ca 00afe724 00afe750 001e114a pjlib_test_i386_Win32_vc14_Debug!pj_test_run+0x98\n00aff988 00215603 00aff9a4 00000001 00dfc4a0 pjlib_test_i386_Win32_vc14_Debug!essential_tests+0x83a\n00affca0 002146b9 00000001 00dfc4a0 00000004 pjlib_test_i386_Win32_vc14_Debug!test_inner+0xe3\n00affd00 001fe881 00000001 00dfc4a0 002a314e pjlib_test_i386_Win32_vc14_Debug!test_main+0x49\n00affd30 00248ae3 00000001 00dfc4a0 00dfe8e0 pjlib_test_i386_Win32_vc14_Debug!main+0x241\n00affd50 00248937 a6ec9253 001e114a 001e114a pjlib_test_i386_Win32_vc14_Debug!invoke_main+0x33\n00affdac 002487cd 00affdbc 00248b68 00affdcc pjlib_test_i386_Win32_vc14_Debug!__scrt_common_main_seh+0x157\n00affdb4 00248b68 00affdcc 7618fcc9 009ff000 pjlib_test_i386_Win32_vc14_Debug!__scrt_common_main+0xd\n00affdbc 7618fcc9 009ff000 7618fcb0 00affe28 pjlib_test_i386_Win32_vc14_Debug!mainCRTStartup+0x8\n00affdcc 770a809e 009ff000 5b0833c3 00000000 kernel32!BaseThreadInitThunk+0x19\n00affe28 770a806e ffffffff 770c9137 00000000 ntdll!__RtlUserThreadStart+0x2f\n00affe38 00000000 001e114a 009ff000 00000000 ntdll!_RtlUserThreadStart+0x1b\n\n\nFAULTING_SOURCE_LINE:  C:\\Users\\bennylp\\Desktop\\project\\pjproject\\pjlib\\src\\pjlib-test\\pool.c\n\nFAULTING_SOURCE_FILE:  C:\\Users\\bennylp\\Desktop\\project\\pjproject\\pjlib\\src\\pjlib-test\\pool.c\n\nFAULTING_SOURCE_LINE_NUMBER:  69\n\nSYMBOL_NAME:  pjlib_test_i386_Win32_vc14_Debug!capacity_test+78\n\nMODULE_NAME: pjlib_test_i386_Win32_vc14_Debug\n\nIMAGE_NAME:  pjlib-test-i386-Win32-vc14-Debug.exe\n\nSTACK_COMMAND:  ~0s ; .ecxr ; kb\n\nFAILURE_BUCKET_ID:  NULL_CLASS_PTR_WRITE_c0000005_pjlib-test-i386-Win32-vc14-Debug.exe!capacity_test\n\nOS_VERSION:  10.0.19041.1\n\nBUILDLAB_STR:  vb_release\n\nOSPLATFORM_TYPE:  x86\n\nOSNAME:  Windows 10\n\nFAILURE_ID_HASH:  {296b5e72-5b8d-2395-c93f-74f7e3cb5d90}\n\nFollowup:     MachineOwner\n---------\n\n\n.  0  Id: 160c.2e90 Suspend: 0 Teb: 00802000 Unfrozen\nChildEBP RetAddr\n00afdc54 7536a823 ntdll!NtWaitForMultipleObjects+0xc\n00afdde8 7536a708 KERNELBASE!WaitForMultipleObjectsEx+0x103\n00afde04 761d800b KERNELBASE!WaitForMultipleObjects+0x18\n00afdeb0 761d7c3c kernel32!WerpReportFaultInternal+0x3b7\n00afdecc 761ad2e9 kernel32!WerpReportFault+0x9d\n00afded4 75428180 kernel32!BasepReportFault+0x19\n00afdf74 770e48a5 KERNELBASE!UnhandledExceptionFilter+0x290\n00affe28 770a806e ntdll!__RtlUserThreadStart+0x3c836\n00affe38 00000000 ntdll!_RtlUserThreadStart+0x1b\n\n   1  Id: 160c.1980 Suspend: 1 Teb: 00805000 Unfrozen\nChildEBP RetAddr\n00dbfd88 77075c30 ntdll!NtWaitForWorkViaWorkerFactory+0xc\n00dbff48 7618fcc9 ntdll!TppWorkerThread+0x2a0\n00dbff58 770a809e kernel32!BaseThreadInitThunk+0x19\n00dbffb4 770a806e ntdll!__RtlUserThreadStart+0x2f\n00dbffc4 00000000 ntdll!_RtlUserThreadStart+0x1b\n\n   2  Id: 160c.1f0 Suspend: 1 Teb: 00808000 Unfrozen\nChildEBP RetAddr\n00fefb60 77075c30 ntdll!NtWaitForWorkViaWorkerFactory+0xc\n00fefd20 7618fcc9 ntdll!TppWorkerThread+0x2a0\n00fefd30 770a809e kernel32!BaseThreadInitThunk+0x19\n00fefd8c 770a806e ntdll!__RtlUserThreadStart+0x2f\n00fefd9c 00000000 ntdll!_RtlUserThreadStart+0x1b\n\n   3  Id: 160c.282c Suspend: 1 Teb: 0080b000 Unfrozen\nChildEBP RetAddr\n0112f6dc 77075c30 ntdll!NtWaitForWorkViaWorkerFactory+0xc\n0112f89c 7618fcc9 ntdll!TppWorkerThread+0x2a0\n0112f8ac 770a809e kernel32!BaseThreadInitThunk+0x19\n0112f908 770a806e ntdll!__RtlUserThreadStart+0x2f\n0112f918 00000000 ntdll!_RtlUserThreadStart+0x1b\nquit:\nNatVis script unloaded from 'C:\\Program Files (x86)\\Windows Kits\\10\\Debuggers\\x64\\Visualizers\\atlmfc.natvis'\nNatVis script unloaded from 'C:\\Program Files (x86)\\Windows Kits\\10\\Debuggers\\x64\\Visualizers\\concurrency.natvis'\nNatVis script unloaded from 'C:\\Program Files (x86)\\Windows Kits\\10\\Debuggers\\x64\\Visualizers\\cpp_rest.natvis'\nNatVis script unloaded from 'C:\\Program Files (x86)\\Windows Kits\\10\\Debuggers\\x64\\Visualizers\\stl.natvis'\nNatVis script unloaded from 'C:\\Program Files (x86)\\Windows Kits\\10\\Debuggers\\x64\\Visualizers\\Windows.Data.Json.natvis'\nNatVis script unloaded from 'C:\\Program Files (x86)\\Windows Kits\\10\\Debuggers\\x64\\Visualizers\\Windows.Devices.Geolocation.natvis'\nNatVis script unloaded from 'C:\\Program Files (x86)\\Windows Kits\\10\\Debuggers\\x64\\Visualizers\\Windows.Devices.Sensors.natvis'\nNatVis script unloaded from 'C:\\Program Files (x86)\\Windows Kits\\10\\Debuggers\\x64\\Visualizers\\Windows.Media.natvis'\nNatVis script unloaded from 'C:\\Program Files (x86)\\Windows Kits\\10\\Debuggers\\x64\\Visualizers\\windows.natvis'\nNatVis script unloaded from 'C:\\Program Files (x86)\\Windows Kits\\10\\Debuggers\\x64\\Visualizers\\winrt.natvis'\n09:51:26 cirunner: Exiting with exit code 3221225477\n```\n\n\u003c/details\u003e\n\n**Analyzing crash dump locally with Visual Studio**\n\nSource level crash debugging is available with VS (tested with VS 2015).\n\nNote:\n\n- make sure your source version is the same as the repository.\n- heap memory values are not available because we use minidump\n\nSteps:\n\n- Download and extract the artifact zip file somewhere (download artifact from [this run](https://github.com/pjsip/pjproject/actions/runs/13150538447?pr=4288) for a sample). It should contain three files: `.exe`, `.dmp`, and `.pdb`.\n- Launch Visual Studio. Open the `.dmp` file\n- Click **Debug with Native only**\n- Find the location of the source file when asked\n\n\n**Limitations/TODO**\n\n1. Maybe we can use **cdb** to generate the crash dump, to avoid using third party **procdump**.\n2. It will be nice if the callstack includes the parameter values like the one produced by **gdb** or **lldb**.\n\n\n## Linux Notes\n\n**Installation**\n\nThe `installlinux.sh` needs to be called once on the target machine to perform the following:\n\n1. install **gdb**\n2. sets `/proc/sys/kernel/core_pattern` to `core.%p`\n\n**Handling crashes**\n\n**cirunner** will take care of setting `ulimit -c` before running the program.\nAny crashes will generate file `core.PID` in the directory where the program is run. \n**cirunner** then runs **gdb** to analyze the crash. See sample output below.\n\n**Handling timeout**\n\n**cirunner** will take care of setting `ulimit -c` before running the program.\nOn timeout, **cirunner** sends `SIGQUIT` which (should) cause the program to crash and generate\ncore. It then runs **gdb** to analyze the minidump.\n\n**Sample crash dump output**\n\nThe output of **gdb** is quite extensive, and very useful. It shows crash location, stack trace, and stack trace\nof all threads, **along with values of all parameters**. Click **Show output** below for sample output with bug induced `pjlib-test` running with `-w 4`.\n\n\u003cdetails\u003e\n  \u003csummary\u003eShow output\u003c/summary\u003e\n\n```\nReading symbols from /home/bennylp/Desktop/project/pjproject/pjlib/bin/pjlib-test-x86_64-unknown-linux-gnu...\n[New LWP 959143]\n[New LWP 959142]\n[New LWP 958137]\n[New LWP 959145]\n[New LWP 959144]\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library \"/lib/x86_64-linux-gnu/libthread_db.so.1\".\nCore was generated by `/home/bennylp/Desktop/project/pjproject/pjlib/bin/pjlib-test-x86_64-unknown-lin'.\nProgram terminated with signal SIGSEGV, Segmentation fault.\n#0  timestamp_test () at ../src/pjlib-test/timestamp.c:139\n139\t    *(long*)(long)rc = 0x1234;\n[Current thread is 1 (Thread 0x7d25bc000640 (LWP 959143))]\n+where\n#0  timestamp_test () at ../src/pjlib-test/timestamp.c:139\n#1  0x00005b4a14f14f9e in run_test_case (runner=0x5b4a1c50c098, tid=2, tc=0x5b4a1516eec0 \u003ctest_app+960\u003e) at ../src/pj/unittest.c:542\n#2  0x00005b4a14f154e7 in text_runner_thread_proc (arg=0x5b4a1c50c258) at ../src/pj/unittest.c:741\n#3  0x00005b4a14ef0d77 in thread_main (param=0x5b4a1c50c268) at ../src/pj/os_core_unix.c:701\n#4  0x00007d25c1c94ac3 in start_thread (arg=\u003coptimized out\u003e) at ./nptl/pthread_create.c:442\n#5  0x00007d25c1d26850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81\n+thread apply all bt\n\nThread 5 (Thread 0x7d25bca00640 (LWP 959144)):\n+bt\n#0  0x00007d25c1ce57f8 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7d25bc9ffd40, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78\n#1  0x00007d25c1cea677 in __GI___nanosleep (req=req@entry=0x7d25bc9ffd40, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25\n#2  0x00007d25c1d1bf2f in usleep (useconds=\u003coptimized out\u003e) at ../sysdeps/posix/usleep.c:31\n#3  0x00005b4a14ef1232 in pj_thread_sleep (msec=100) at ../src/pj/os_core_unix.c:948\n#4  0x00005b4a14f154fc in text_runner_thread_proc (arg=0x5b4a1c50c328) at ../src/pj/unittest.c:746\n#5  0x00005b4a14ef0d77 in thread_main (param=0x5b4a1c50c338) at ../src/pj/os_core_unix.c:701\n#6  0x00007d25c1c94ac3 in start_thread (arg=\u003coptimized out\u003e) at ./nptl/pthread_create.c:442\n#7  0x00007d25c1d26850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81\n\nThread 4 (Thread 0x7d25bd400640 (LWP 959145)):\n+bt\n#0  0x00007d25c1ce57f8 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7d25bd3ffd40, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78\n#1  0x00007d25c1cea677 in __GI___nanosleep (req=req@entry=0x7d25bd3ffd40, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25\n#2  0x00007d25c1d1bf2f in usleep (useconds=\u003coptimized out\u003e) at ../sysdeps/posix/usleep.c:31\n#3  0x00005b4a14ef1232 in pj_thread_sleep (msec=100) at ../src/pj/os_core_unix.c:948\n#4  0x00005b4a14f154fc in text_runner_thread_proc (arg=0x5b4a1c50c3f8) at ../src/pj/unittest.c:746\n#5  0x00005b4a14ef0d77 in thread_main (param=0x5b4a1c50c408) at ../src/pj/os_core_unix.c:701\n#6  0x00007d25c1c94ac3 in start_thread (arg=\u003coptimized out\u003e) at ./nptl/pthread_create.c:442\n#7  0x00007d25c1d26850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81\n\nThread 3 (Thread 0x7d25c24bd540 (LWP 958137)):\n+bt\n#0  0x00007d25c1ce57f8 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fff1cca6e50, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78\n#1  0x00007d25c1cea677 in __GI___nanosleep (req=req@entry=0x7fff1cca6e50, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25\n#2  0x00007d25c1d1bf2f in usleep (useconds=\u003coptimized out\u003e) at ../sysdeps/posix/usleep.c:31\n#3  0x00005b4a14ef1232 in pj_thread_sleep (msec=100) at ../src/pj/os_core_unix.c:948\n#4  0x00005b4a14f154fc in text_runner_thread_proc (arg=0x7fff1cca6f10) at ../src/pj/unittest.c:746\n#5  0x00005b4a14f15591 in text_runner_main (base=0x5b4a1c50c098) at ../src/pj/unittest.c:770\n#6  0x00005b4a14f1445c in pj_test_run (runner=0x5b4a1c50c098, suite=0x5b4a1516eb30 \u003ctest_app+48\u003e) at ../src/pj/unittest.c:235\n#7  0x00005b4a14ee5300 in ut_run_tests (ut_app=0x5b4a1516eb00 \u003ctest_app\u003e, title=0x5b4a14f1e84e \"features tests\", argc=1, argv=0x7fff1cca79f8) at ../src/pjlib-test/test_util.h:202\n#8  0x00005b4a14ee6734 in features_tests (argc=1, argv=0x7fff1cca79f8) at ../src/pjlib-test/test.c:350\n#9  0x00005b4a14ee68e2 in test_inner (argc=1, argv=0x7fff1cca79f8) at ../src/pjlib-test/test.c:397\n#10 0x00005b4a14ee6a70 in test_main (argc=1, argv=0x7fff1cca79f8) at ../src/pjlib-test/test.c:445\n#11 0x00005b4a14ebced7 in main (argc=1, argv=0x7fff1cca79f8) at ../src/pjlib-test/main.c:172\n\nThread 2 (Thread 0x7d25bb600640 (LWP 959142)):\n+bt\n#0  0x00007d25c1ce57f8 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7d25bb5ffd40, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78\n#1  0x00007d25c1cea677 in __GI___nanosleep (req=req@entry=0x7d25bb5ffd40, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25\n#2  0x00007d25c1d1bf2f in usleep (useconds=\u003coptimized out\u003e) at ../sysdeps/posix/usleep.c:31\n#3  0x00005b4a14ef1232 in pj_thread_sleep (msec=100) at ../src/pj/os_core_unix.c:948\n#4  0x00005b4a14f154fc in text_runner_thread_proc (arg=0x5b4a1c50c188) at ../src/pj/unittest.c:746\n#5  0x00005b4a14ef0d77 in thread_main (param=0x5b4a1c50c198) at ../src/pj/os_core_unix.c:701\n#6  0x00007d25c1c94ac3 in start_thread (arg=\u003coptimized out\u003e) at ./nptl/pthread_create.c:442\n#7  0x00007d25c1d26850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81\n\nThread 1 (Thread 0x7d25bc000640 (LWP 959143)):\n+bt\n#0  timestamp_test () at ../src/pjlib-test/timestamp.c:139\n#1  0x00005b4a14f14f9e in run_test_case (runner=0x5b4a1c50c098, tid=2, tc=0x5b4a1516eec0 \u003ctest_app+960\u003e) at ../src/pj/unittest.c:542\n#2  0x00005b4a14f154e7 in text_runner_thread_proc (arg=0x5b4a1c50c258) at ../src/pj/unittest.c:741\n#3  0x00005b4a14ef0d77 in thread_main (param=0x5b4a1c50c268) at ../src/pj/os_core_unix.c:701\n#4  0x00007d25c1c94ac3 in start_thread (arg=\u003coptimized out\u003e) at ./nptl/pthread_create.c:442\n#5  0x00007d25c1d26850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81\n+quit\n10:38:27 cirunner: Exiting with exit code -11\n```\n\n\u003c/details\u003e\n\n\n**Analyzing crash dump locally with VS Code**\n\nSource level crash debugging is available with VS Code.\n\nNote:\n\n- make sure your source file version is the same as the repository.\n- for Linux, heap memory values are not available\n\nDownload the artifact zip to a linux machine, extract somewhere (e.g. download sample artifact from [this Linux run](https://github.com/pjsip/pjproject/actions/runs/13150538445?pr=4288)).\n\nUse the following as template for your `launch.json`. The important keys are:\n\n- `program`\n- `coreDumpPath`\n- `sourceFileMap`\n\n```\n    {\n        \"name\": \"coredump\",\n        \"type\": \"cppdbg\",\n        \"request\": \"launch\",\n        \"program\": \"/path/to/artifact/pjsip-test-x86_64-unknown-linux-gnu\",\n        \"coreDumpPath\": \"/path/to/artifact//core.6469\",\n        \"cwd\": \"does not matter?\",\n        \"sourceFileMap\": {\n            \"/home/runner/work/pjproject/pjproject\": \"/your/source/to/project/pjproject\"\n        },\n        \"environment\": [\n            {\n                \"name\": \"LD_LIBRARY_PATH\",\n                \"value\": \"/additional/dll/such/as/openh264\"\n            }\n        ]\n    },\n```\n\n**Analyzing crash dump locally with gdb**\n\n1. download the artifact zip to a linux machine, extract somewhere (e.g. download sample artifact from [this run](https://github.com/pjsip/pjproject/actions/runs/13150538445?pr=4288))\n2. if the crashed program is **pjlib-test**, change directory to `your/source/pjproject/pjlib/bin`, \n   if the crashed program is **pjsip-test**, change directory to `your/source/pjproject/pjsip/bin`, \n   etc.\n3. run **gdb**:\n   ```\n   $ gdb /path/to/pjsip-test /path/to/core.12345\n   ```\n\n\n## MacOS Notes\n\n**Installation**\n\nThe `installmac.sh` doesn't do anything. We use `lldb` which is already installed\nand existing core pattern `/cores/core.%P`.\n\n**Handling crashes**\n\n**cirunner** will take care of setting `ulimit -c` before running the program.\nAny crashes will generate file `core.PID` in `/cores` directory. \n**cirunner** then runs **lldb** to analyze the crash. See sample output below.\n\n**Handling timeout**\n\n**cirunner** will take care of setting `ulimit -c` before running the program.\nOn timeout, **cirunner** sends `SIGQUIT` which (should) cause the program to crash and generate\ncore. It then runs **lldb** to analyze the minidump.\n\n**Sample crash dump output**\n\n**lldb** output is okay, it shows crash location, stack trace, and stack trace\nof all threads, **along with values of all parameters**.\nClick **Show output** below for sample output with bug induced `pjlib-test` running with `-w 4`.\n\n\u003cdetails\u003e\n  \u003csummary\u003eShow output\u003c/summary\u003e\n\n```\n(lldb) target create \"/Users/runner/work/pjproject/pjproject/pjlib/bin/pjlib-test-arm-apple-darwin23.6.0\" --core \"/cores/core.18011\"\nCore file '/cores/core.18011' (arm64) was loaded.\n(lldb) bt all\nwarning: could not execute support code to read Objective-C class data in the process. This may reduce the quality of type information available.\n\n* thread #1, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x0)\n  * frame #0: 0x0000000100e64ef0 pjlib-test-arm-apple-darwin23.6.0`timestamp_test at timestamp.c:139:22\n  thread #2\n    frame #0: 0x00000001932643c8 libsystem_kernel.dylib`__semwait_signal + 8\n    frame #1: 0x0000000193145568 libsystem_c.dylib`nanosleep + 220\n    frame #2: 0x0000000193145480 libsystem_c.dylib`usleep + 68\n    frame #3: 0x0000000100e6eb60 pjlib-test-arm-apple-darwin23.6.0`pj_thread_sleep(msec=100) at os_core_unix.c:948:5\n    frame #4: 0x0000000100e9ef90 pjlib-test-arm-apple-darwin23.6.0`text_runner_thread_proc(arg=0x00000001378134f0) at unittest.c:746:13\n    frame #5: 0x0000000100e6e5dc pjlib-test-arm-apple-darwin23.6.0`thread_main(param=0x0000000137813500) at os_core_unix.c:701:27\n    frame #6: 0x00000001932a1f94 libsystem_pthread.dylib`_pthread_start + 136\n  thread #3\n    frame #0: 0x00000001932643c8 libsystem_kernel.dylib`__semwait_signal + 8\n    frame #1: 0x0000000193145568 libsystem_c.dylib`nanosleep + 220\n    frame #2: 0x0000000193145480 libsystem_c.dylib`usleep + 68\n    frame #3: 0x0000000100e6eb60 pjlib-test-arm-apple-darwin23.6.0`pj_thread_sleep(msec=100) at os_core_unix.c:948:5\n    frame #4: 0x0000000100e9ef90 pjlib-test-arm-apple-darwin23.6.0`text_runner_thread_proc(arg=0x00000001378135d8) at unittest.c:746:13\n    frame #5: 0x0000000100e6e5dc pjlib-test-arm-apple-darwin23.6.0`thread_main(param=0x00000001378135e8) at os_core_unix.c:701:27\n    frame #6: 0x00000001932a1f94 libsystem_pthread.dylib`_pthread_start + 136\n(lldb) quit\n08:50:39 cirunner: Exiting with exit code -11\n```\n\n\u003c/details\u003e\n\n\n**TODO**\n\nMaybe someone can improve **lldb** commands to dump more info about the crash (similar to `where`\ncommand in **gdb**).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpjsip%2Fcirunner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpjsip%2Fcirunner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpjsip%2Fcirunner/lists"}