{"id":21594687,"url":"https://github.com/ispras/juliet-dynamic","last_synced_at":"2026-02-09T04:34:40.882Z","repository":{"id":72016760,"uuid":"407553992","full_name":"ispras/juliet-dynamic","owner":"ispras","description":"Juliet C/C++ Dynamic Test Suite","archived":false,"fork":false,"pushed_at":"2023-04-18T15:41:53.000Z","size":40551,"stargazers_count":29,"open_issues_count":2,"forks_count":7,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-08-01T00:12:02.872Z","etag":null,"topics":["binary","binary-analysis","checkers","cwe","dynamic","error-detection","juliet","symbolic-execution","testsuite"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ispras.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-17T13:39:07.000Z","updated_at":"2025-04-29T02:43:14.000Z","dependencies_parsed_at":null,"dependency_job_id":"51b18063-13d1-464b-b6a7-f3ed4a426987","html_url":"https://github.com/ispras/juliet-dynamic","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ispras/juliet-dynamic","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ispras%2Fjuliet-dynamic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ispras%2Fjuliet-dynamic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ispras%2Fjuliet-dynamic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ispras%2Fjuliet-dynamic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ispras","download_url":"https://codeload.github.com/ispras/juliet-dynamic/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ispras%2Fjuliet-dynamic/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29256709,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-09T04:11:57.159Z","status":"ssl_error","status_checked_at":"2026-02-09T04:11:56.117Z","response_time":56,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binary","binary-analysis","checkers","cwe","dynamic","error-detection","juliet","symbolic-execution","testsuite"],"created_at":"2024-11-24T17:19:19.462Z","updated_at":"2026-02-09T04:34:40.867Z","avatar_url":"https://github.com/ispras.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Juliet C/C++ Dynamic Test Suite\n\nThis repository contains [Juliet](https://samate.nist.gov/SRD/testsuite.php)\nC/C++ test suite for dynamic tools. You can measure True Positive and True\nNegative rates for your dynamic tool. This repository was used to measure\nresults for \"Symbolic Security Predicates: Hunt Program Weaknesses\"\n[paper](https://arxiv.org/abs/2111.05770). However, feel free to support your\nown tool.\n\n## Dependencies\n\n    $ sudo apt install clang-10 gcc-multilib\n\n## Build\n\nBuilding all CWEs may require a long time. Building required CWE to measure\nSydr:\n\n    $ CC=clang-10 CXX=clang++-10 make -j15 CWE121 CWE122 CWE124 CWE126 CWE127 CWE190 CWE191 CWE194 CWE195 CWE369 CWE680\n\n## Usage\n\n    usage: test_juliet.py [-h]\n                          [-c [CWE_NUM_1 CWE_NUM_2 ... [CWE_NUM_1 CWE_NUM_2 ... ...]]]\n                          [-t TOOL] [-e] [-d] [-r] [-j THREADS]\n\n    optional arguments:\n      -h, --help            show this help message and exit\n      -c [CWE_NUM_1 CWE_NUM_2 ... [CWE_NUM_1 CWE_NUM_2 ... ...]], --cwe [CWE_NUM_1 CWE_NUM_2 ... [CWE_NUM_1 CWE_NUM_2 ... ...]]\n                            Run specified CWE.\n      -t TOOL, --tool TOOL  Tool name.\n      -e, --error           Print false positive and false negative tests.\n      -d, --delete          Delete results and collect them again.\n      -r, --reproduce       Recalculate statistics and reproduce sanitizers\n                            verification.\n      -j THREADS            Set number of threads.\n\n## Run Juliet\n\nYou can measure TPR and TNR for your tool via `test_juliet.py` script, which\ndoes the following:\n\n1. The script runs your tool on all Juliet test cases.\n2. Each test case is executed on a valid input that does not lead to an error.\n3. Your tool should generate new inputs that may trigger an error.\n4. The script assigns TP to positive test case if at least one input was\n   generated. The script assigns FP to negative test case if at least one input\n   was generated. The script assigns TN to negative test case if no inputs were\n   generated. The script assigns FN to positive test case if no inputs were\n   generated.\n5. Afterwards, the script verifies TP cases on sanitizers. TP stays for test\n   case if sanitizers signal an error for at least one generated input, FN is\n   assigned to the test case otherwise.\n\nThe following script runs Sydr on all Juliet test cases, determines classes (TP,\nFP, TN, FN) for each test case, and verifies generated inputs on sanitizers:\n\n    $ ./test_juliet.py -j4\n\nThe script will generate `results/stats.json` file containing classes (TP, FP,\nTN, FN) and sanitizers verification result for each checked Juliet test case.\n\nMoreover, script prints overall TPR, TNR, and\n[ACC](https://en.wikipedia.org/wiki/Sensitivity_and_specificity) before/after\nsanitizers verification.\n\nThe sequential runs of `test_juliet.py` will only print the already obtained\nstatistics.\n\nThe following command prints all False Positive and False Negative cases:\n\n    $ ./test_juliet.py -e\n\n## Generating LaTex Table\n\nSave statistics to file (results must be already collected):\n\n    $ ./test_juliet.py \u003e stats.txt\n\nGenerate LaTex file:\n\n    $ ./table.py \u003e stats.tex\n\nBuilding pdf:\n\n    $ pdflatex stats.tex\n\nGenerating svg:\n\n    $ pdf2svg stats.pdf stats.svg\n\n## Reproducing Statistics\n\nIf you do not acquire Sydr, you can reproduce statistics from raw results:\n\n1. Extract results archive in Juliet root:\n```\n$ tar xf results.tar.xz\n```\n2. The following script will remove `results/stats.json` and rerun sanitizers\n   verification:\n```\n$ ./test_juliet.py --reproduce -j4\n```\n\nIf you want to generate LaTex table, see section above.\n\n## Supporting Your Dynamic Tool\n\nYou just need to implement `run_TOOLNAME` function in `test_juliet.py`. This\nfunction accepts path to a single Juliet test case binary, CWE enum for the test\ncase, and path to corresponding input file containing stdin. The function should\nreturn list of new generated inputs by your tool. Then just run:\n\n    $ ./test_juliet.py -j4 -t TOOLNAME\n\n## Sydr Evaluation\n\nWe evaluated Sydr security predicates on Juliet test suite. See results below:\n\n![Sydr Juliet results](stats.svg)\n\n## Cite Us\n\n```\n@article{vishnyakov21,\n  title = {Symbolic Security Predicates: Hunt Program Weaknesses},\n  author = {Vishnyakov, Alexey and Logunova, Vlada and Kobrin, Eli and Kuts,\n            Daniil and Parygina, Darya and Fedotov, Andrey},\n  booktitle = {2021 Ivannikov ISPRAS Open Conference (ISPRAS)},\n  year = {2021},\n  publisher = {IEEE},\n  url = {https://arxiv.org/abs/2111.05770},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fispras%2Fjuliet-dynamic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fispras%2Fjuliet-dynamic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fispras%2Fjuliet-dynamic/lists"}