{"id":21404698,"url":"https://github.com/checkedc/checkedc-llvm-test-suite","last_synced_at":"2026-01-03T07:31:17.375Z","repository":{"id":179569800,"uuid":"663374493","full_name":"checkedc/checkedc-llvm-test-suite","owner":"checkedc","description":null,"archived":false,"fork":false,"pushed_at":"2023-07-29T19:00:32.000Z","size":515875,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-23T03:33:23.667Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/checkedc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.TXT","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-07T06:43:48.000Z","updated_at":"2023-07-07T06:51:55.000Z","dependencies_parsed_at":"2025-01-23T03:43:02.800Z","dependency_job_id":null,"html_url":"https://github.com/checkedc/checkedc-llvm-test-suite","commit_stats":null,"previous_names":["checkedc/checkedc-llvm-test-suite"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/checkedc%2Fcheckedc-llvm-test-suite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/checkedc%2Fcheckedc-llvm-test-suite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/checkedc%2Fcheckedc-llvm-test-suite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/checkedc%2Fcheckedc-llvm-test-suite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/checkedc","download_url":"https://codeload.github.com/checkedc/checkedc-llvm-test-suite/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243902292,"owners_count":20366259,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-22T16:17:37.766Z","updated_at":"2026-01-03T07:31:12.354Z","avatar_url":"https://github.com/checkedc.png","language":"C","readme":"# The Checked C LLVM test-suite repo\n\nThis repo contains a version of the LLVM test-suite repo that is being modified\nto use Checked C. The modified programs will be used to benchmark the Checked C\nversion of LLVM/clang.\n\nWe have deleted test-only code from the main branch of the repo and left only\nbenchmarks in the main branch.  That makes the repo easier to work with.  It\ndecreases disk usage from about 2.3 GBytes to under 500 MBytes when using the\nmain branch.\n\nChecked C is an extension to C that adds checking to detect or prevent common \nprogramming  errors such as out-of-bounds memory accesses.  For more information\non Checked C, see the Checked C specification in the\n[Checked C repo](https://github.com/Microsoft/checkedc).  The Checked C\nversion of LLVM/clang lives in two repos: the\n[Checked C clang repo](https://github.com/Microsoft/checked-clang)\nand the [Checked C LLVM repo](https://github.com/Microsoft/checkedc-llvm).\n\n\n## Branch organization\n\nThere are 3 branches in the repo:\n- main: this branch contains benchmarks, some of which may have been modified\nto use Checked C.\n- baseline: this branch contains benchmarks that have not been modified.\n- original: this contains all the tests, including application tests.\n\nThis main branch should be used for modifying benchmarks.  This branch can be diffed\nagainst the baseline branch to see the changes in benchmarks.\nThe original branch can be used to test that\nthe Checked C implementation has not broken existing tests.\n\n## Running tests\n\nThis has been adapted from the `test-suite`\n[Quickstart guide](https://llvm.org/docs/TestSuiteGuide.html).\n\nPrerequisite: Make sure you have checked out and\n[built](https://github.com/checkedc/checkedc-llvm-project/blob/main/clang/docs/checkedc/Setup-and-Build.md)\nthe Checked C compiler.\n\n1. The lit test runner is required to run the tests. You can use the one from\nyour Checked C compiler build:\n```\n% \u003cpath to llvm build\u003e/bin/llvm-lit --version\nlit 0.8.0dev\n```\n2. Check out the test-suite repo:\n```\n git clone https://github.com/checkedc/checkedc-llvm-test-suite.git test-suite\n ```\n\n 3. Create a build directory and use CMake to configure the suite. Use the\n `CMAKE_C_COMPILER`` option to specify the compiler to test. Use a cache file\nto choose a typical build configuration:\n```\n% mkdir test-suite-build\n% cd test-suite-build\n% cmake -DCMAKE_C_COMPILER=\u003cpath to llvm build\u003e/bin/clang \\\n        -C../test-suite/cmake/caches/Debug.cmake \\\n        ../test-suite\n```\nTo run tests with optmizations enabled, use\n```\n-C../test-suite/cmake/caches/O3.cmake\n```\ninstead of Debug.cmake\n\n4. Build the benchmarks:\n```\n% make\nScanning dependencies of target timeit-target\n[  0%] Building C object tools/CMakeFiles/timeit-target.dir/timeit.c.o\n[  0%] Linking C executable timeit-target\n...\n```\n5. Run the tets with lit:\n```\n% \u003cpath to llvm build\u003e/bin/llvm-lit -v -j 1 -o results.json .\n```\n\n## Deprecated approach using LNT to run tests\n\nWe do not recommend running tests using LNT because LNT is not well-maintained.\nIt use very old packages that use features deprecated and removed from the core \nPython frameworks as of Python 3.10.\n\n### On Linux\n1. Setup LNT\nNote: These steps have been adopted from the\n[LNT Quickstart Guide](http://llvm.org/docs/lnt/quickstart.html).\nThese instructions are for Ubuntu 20.\n```\nsudo apt install bison flex tclsh\nsudo apt install virtualenv\nsudo virtualenv ~/mysandbox\ngit clone https://github.com/llvm/llvm-lnt.git  ~/lnt\nsudo ~/mysandbox/bin/python ~/lnt/setup.py develop\n```\n\n2. Invoke LNT tests\n\nPrerequisite: Make sure you have checked out and\n[built](https://github.com/checkedc/checkedc-llvm-project/blob/main/clang/docs/checkedc/Setup-and-Build.md)\nthe Checked C compiler.\n```\ngit clone https://github.com/checkedc/workfilow.git \u003cAUTOMATION_DIR\u003e\nexport SRC_DIR=\u003c/path/to/llvm/src\u003e\nexport BUILD_DIR=\u003c/path/to/llvm/build\u003e\nexport CLANG_REPO=https://github.com/checkedc/checkedc-llvm-project.git\nexport CHECKEDC_REPO=https://github.com/checkedc/checkedc.git\n\u003cAUTOMATION_DIR\u003e/UNIX/run-lnt-local.sh\n```\n\nOptional flags:\n```\nTEST_TARGET=\"X86_64;ARM\"\nLNT_BIN=\u003c/path/to/lnt\u003e // By default, lnt is picked up from ~/mysandbox/bin/lnt.\n```\n\nThe test results are generated at:\n```\n\u003cBUILD_DIR\u003e/LNT-Results-Release-Linux/\u003cTEST_TARGET\u003e/test-\u003cTIME_STAMP\u003e/test.log\n```\n\n### On MacOS\n\nThis has been tested on MacOS Ventura 13.4.1 on Apple Silicon.\n\n1. Setup LNT\n- Install Python 3.9 or earlier.\n- Run\n```\n   python3.9 venv ~/mysandbox\n   git clone https://github.com/llvm/llvm-lnt.git  ~/lnt\n   ~/mysandbox/bin/python ~/lnt/setup.py develop\n```\n It will fail to install.  Change director to your sandbox, install the\n following packages, and re-run the LNTinstall script\n```\ncd ~/mysandbox\nbin/pip install pyyaml==5.1.2\nbin/pip install SQLAlchemy==1.2.19\nbin/pip install MarkupSafe==0.23\nbin/python ~lnt/setup.py develope\n```\n2. Invoke LNT tests\nFollow the directions in step 2 for Linux.\n\n### On Windows\n\nThe LNT tests can also be run on Windows 10 using\nthe [Windows Subsystem for Linux](https://blogs.msdn.microsoft.com/wsl/2016/04/22/windows-subsystem-for-linux-overview/).\nSee the directions [here](docs/Benchmarking-on-Windows.md).\n\n\n\n## Contributing\n\nWe would be happy for people to convert existing benchmarks to use Checked C.\nFor code contributions, we follow the standard\n[Github workflow](https://guides.github.com/introduction/flow/).  See \n[Contributing to Checked C](https://github.com/checkekdc/checkedc/blob/main/CONTRIBUTING.md) for more detail.\nYou will need to sign a contributor license agreement before contributing a\nconverted benchmark.\n\nFor more information on contributing on the Checked C project, see \n[Contributing to Checked C](https://github.com/checkedc/checkedc/blob/main/CONTRIBUTING.md).\n\n## Code of conduct\n\nThis project has adopted a\n[Code of Conduct](https://github.com/checkedc/checkedc/blob/main/CODE_OF_CONDUCT.md).\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcheckedc%2Fcheckedc-llvm-test-suite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcheckedc%2Fcheckedc-llvm-test-suite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcheckedc%2Fcheckedc-llvm-test-suite/lists"}