{"id":16659084,"url":"https://github.com/agroce/onward24code","last_synced_at":"2025-06-10T20:35:05.190Z","repository":{"id":235892347,"uuid":"790292682","full_name":"agroce/onward24code","owner":"agroce","description":"Source code and instructions for \"(Programs), Proofs and Refutations (and Tests and Mutants)\"","archived":false,"fork":false,"pushed_at":"2024-08-07T19:05:50.000Z","size":16,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-12T20:36:52.874Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/agroce.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-22T16:04:31.000Z","updated_at":"2024-08-07T19:05:54.000Z","dependencies_parsed_at":"2024-11-21T01:15:52.154Z","dependency_job_id":null,"html_url":"https://github.com/agroce/onward24code","commit_stats":null,"previous_names":["agroce/onward24code"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agroce%2Fonward24code","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agroce%2Fonward24code/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agroce%2Fonward24code/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agroce%2Fonward24code/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/agroce","download_url":"https://codeload.github.com/agroce/onward24code/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/agroce%2Fonward24code/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259146557,"owners_count":22812261,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-12T10:23:43.927Z","updated_at":"2025-06-10T20:35:05.163Z","avatar_url":"https://github.com/agroce.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"This repository contains code and (simple) instructions for \"following along\" with the ideas\npresented in the dialogue.\n\nThe easiest way to work with this code is to use a docker image with DeepState, CBMC (an old version, admittedly), and UniversalMutator already installed:\n\n```\ndocker pull agroce/onward24_docker\ndocker run -it agroce/onward24_docker\ncd ~/onward24code\n```\n\nThe image already has mutants generated for `binsearch.c` in the `mutants` directory.  You can generate them yourself:\n\n```\nmutate binsearch.c --cmd \"clang -c binsearch.c\" --mutantDir mutants\n```\n\nThis invokes the \"bugginator\" to produce over 80 likely-buggy versions of `binsearch.c`.  The `--cmd` option tells the tool to check that the mutant is valid C code by trying to compile it.\n\nTo check the code using cbmc, just type:\n\n```\ncbmc binsearch_cbmc.c binsearch.c --unwind 12 --bounds-check --pointer-check --unwinding-assertions\n```\n\nIf you alter 'MAX_SIZE` you'll need to alter the unwinding depth, too.\n\nUsing DeepState is a bit more complex.  First you have to build the various test executables.  The included Makefile does this for you, however.  So, to run one test (the \"default test\") of binary search, you just type:\n\n```\nmake\n./test_binsearch\n```\n\nYou should see something like:\n\n```\nTRACE: Running: Run_Bentley from binsearch_deepstate.cpp(10)\nTRACE: binsearch_deepstate.cpp(16): SIZE = 1\nTRACE: binsearch_deepstate.cpp(20): a[0] = 0\nTRACE: binsearch_deepstate.cpp(25): Sorting...\nTRACE: binsearch_deepstate.cpp(28): a[0] = 0\nTRACE: binsearch_deepstate.cpp(35): k = 0\nTRACE: binsearch_deepstate.cpp(36): present = 1\nTRACE: binsearch_deepstate.cpp(39): r = 0\nTRACE: Passed: Run_Bentley\n```\n\nTo use DeepState's built-in (very dumb, not coverage-driven) fuzzer:\n\n```\n./test_binsearch --fuzz --timeout 30\n```\n\nYou'll see something like:\n\n```\nINFO: Starting fuzzing\nWARNING: No seed provided; using 1714058690\nWARNING: No test specified, defaulting to first test defined (Run_Bentley)\nINFO: Done fuzzing! Ran 1391568 tests (46385 tests/second) with 0 failed/1391568 passed/0 abandoned tests\n```\n\nFinally, to fuzz the binary search using AFL++, a very good coverage-driven mutation-based fuzzer:\n\n```\ndeepstate-afl ./fuzz_binsearch -o fuzzing_output --fuzzer_out\n```\n\nStop the fuzzer with Ctrl-C.\n\nFinally, how do you check the mutants?\n\nFor cbmc, it's easy:\n\n```\nanalyze_mutants binsearch.c \"cbmc binsearch_cbmc.c binsearch.c --unwind 12 --bounds-check --pointer-check --unwinding-assertions\" --timeout 600 --verbose --mutantDir mutants\n```\n\nThis will take some time!\n\nTo look at the mutants not detected (thus possible holes in the specification or harness):\n\n```\nshow_mutants unkilled.txt --mutantDir mutants\n```\n\nThe same approach will work with DeepState:\n\n```\nanalyze_mutants binsearch.c \"make clean; make test_binsearch; ./test_binsearch --fuzz --timeout 15 --abort_on_fail\" --timeout 20 --verbose --mutantDir mutants\n```\n\nBoth CBMC and DeepState should detect more than 80% of the mutants as\nfaulty, suggesting we have a fairly strong specification.  DeepState's\nresults will vary, due to the nature of random value generation.  CBMC\nshould consistently detect almost 93% of  the mutants: when it is\npossible, proof can be more powerful than testing.  The undetected\nmutants in some cases are genuinely equivalent.  The one other case\nconverts binary search into a very strange linear search with\nadditional requirements, suggesting the omission of performance\ntesting is a serious issue with our approach.\n\nThe (undocumented, at least for now) code in the `advanced` directory\nshows a start on trying to verify binary search, including\nperformance, for unbounded array sizes (up to index type size in any\ncase), thus covering the Bloch case also.\n\n**ADDENDUM**\n\nYou can also try binary-based mutation using [MuttFuzz](https://github.com/agroce/muttfuzz).  One nice thing with this approach is that the problem of equivalent mutants --\nmutants that change source code but not meaningful program semantics -- is much less of an issue when mutants all involve changing\nbinary-level jumps in a program.  Change a reachale jump instruction and you probably have a seriously different problem!\n\nTo play with this:\n\n```\npip install muttfuzz --upgrade\nmuttfuzz \"rm -rf fuzz; deepstate-afl ./fuzz_binsearch -o fuzz --timeout 30; ./fuzz_binsearch --input_test_files_dir fuzz/the_fuzzer/crashes/ --abort_on_fail \u0026\u0026 ./fuzz_binsearch --input_test_files_dir fuzz/the_fuzzer/queue/ --abort_on_fail \u0026\u0026 ./fuzz_binsearch --input_test_files_dir fuzz/the_fuzzer/hangs/ --abort_on_fail\" ./fuzz_binsearch --score --time_per_mutant 80 --source_only_mutate binsearch --avoid_repeats --stop_on_repeat\n```\n\nThe upgrade is there because muttfuzz is more of a work-in-progress than the other tools, so grabbing the latest before using it is a good idea.  This will actually run through all the binary-level jump mutants of the binsearch code.  If you are curious about exploring the mutants more, create a directory and add `--save_mutants \u003cdir\u003e` to the muttfuzz arguments.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fagroce%2Fonward24code","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fagroce%2Fonward24code","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fagroce%2Fonward24code/lists"}