{"id":17267475,"url":"https://github.com/technius/soltype-artifact-public","last_synced_at":"2025-08-17T21:15:24.136Z","repository":{"id":143021378,"uuid":"414506329","full_name":"Technius/soltype-artifact-public","owner":"Technius","description":null,"archived":false,"fork":false,"pushed_at":"2021-10-26T04:24:06.000Z","size":599,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-26T11:14:07.924Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Solidity","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Technius.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-07T07:42:18.000Z","updated_at":"2021-10-18T00:17:37.000Z","dependencies_parsed_at":"2023-03-26T17:33:35.056Z","dependency_job_id":null,"html_url":"https://github.com/Technius/soltype-artifact-public","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/Technius/soltype-artifact-public","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Technius%2Fsoltype-artifact-public","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Technius%2Fsoltype-artifact-public/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Technius%2Fsoltype-artifact-public/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Technius%2Fsoltype-artifact-public/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Technius","download_url":"https://codeload.github.com/Technius/soltype-artifact-public/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Technius%2Fsoltype-artifact-public/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270907429,"owners_count":24665962,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-17T02:00:09.016Z","response_time":129,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-15T08:10:45.124Z","updated_at":"2025-08-17T21:15:24.081Z","avatar_url":"https://github.com/Technius.png","language":"Solidity","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SolType artifact\n\nTo facilitate artifact evaluation, we have packaged our binaries as a docker\nimage, with the supporting benchmarks and logs stored in this directory. You can\nretrieve the docker image using the following command:\n\n```bash\ndocker pull technius/soltype-artifact:latest\n```\n\nWe describe some of the supporting files below:\n* `run_benchmarks.sh`: generates output from Solid on the representative\n  benchmark files.\n* `solc_vers.csv`: contains solidity compiler version information\n* `cinv.csv`: lists the contract invariant associated with each contract\n* `author-logs/`: contains log files that were generated by the authors\n* `data/`: contains the representative benchmark files\n\nDue to the large number of benchmark files in our evaluation, we think it will\nbe too time consuming to verify the claims for all 120 benchmarks. Thus, we have\nselected 5 representative benchmarks for which be believe the claims can be\nverified within a reasonable amount of time. We are happy to provide more\nbenchmarks upon request.\n\n## Setup and Usage\n\nIn a shell, navigate to this directory.\n\nThen, launch the prepackaged environment using docker:\n\n```bash\nsudo docker run --rm -it -v \"$PWD\":/opt/eval technius/soltype-artifact:latest\n```\n\nThis will mount the artifact data folder into the image and then place you\ninto a bash shell. Make sure you are able to locate `data/sanity_test.sol`\nusing the `ls` command.\n\nNow, check that the `solid` command produces help output:\n\n```bash\nsolid -h\n```\n\nIn general, AutoSolid can be run using the format:\n\n```bash\nsolid --solc \u003cpath_to_solc\u003e --only-last \u003ccontract .sol file\u003e\n```\n\nThis will run Solid on the last (concrete) contract contained in the provided\n`.sol` file.\n\nSemiSolid can be run using:\n\n```bash\nsolid --solc \u003cpath_to_solc\u003e --only-last \u003ccontract name\u003e --task check -i \u003ccontract invariant\u003e\n```\n\nTwo versions of `solc` are available in the image: `solc_0.4.26` and `solc_0.5.17`.\n\n### AutoSolid\n\nTo ensure that everything works correctly, try the sanity test:\n\n```bash\n$ solid --solc solc_0.4.26 data/sanity_test.sol\n\nNow running on SanityTest\n\nBootstrapping...\nTrying templates...\n[]\nend templates\nIteration 0\nsafe: [8,3,2]\nList [Atom \"not\",List [Atom \"\u003e=\",Atom \"x!0\",Atom \"101\"]]\nTotal math ops: 3\nProvably safe math ops: 3\nInferred contract invariant: (not (\u003e= x 101 ) )\nThe following safe math checks are redundant:\nbar: line 14: {v:bool | v == ((x$3 + (cast[uint256](5))) \u003c= 115792089237316195423570985008687907853269984665640564039457584007913129639935)} ==\u003e {v:bool | v == true}\nfoo: line 10: {v:bool | v == ((x$2 + a$0) \u003c= 115792089237316195423570985008687907853269984665640564039457584007913129639935)} ==\u003e {v:bool | v == true}\nfoo: line 8: {v:bool | v == ((x$1 + 1) \u003c= 115792089237316195423570985008687907853269984665640564039457584007913129639935)} ==\u003e {v:bool | v == true}\nThe following safe math checks are necessary:\nSolving time: 0.030567557s\n```\n\nThis will run Solid in automated mode. The output is divided into the following\nsections:\n\n1. \"Live\" output tracking when a query is UNSAT or times out. Queries that time\n   out will be retried up to 3 times.\n2. Once the algorithm finishes, the inferred contract invariant will be\n   displayed (`Inferred contract invariant: ...`). As Solid is still a\n   prototype, the invariant is displayed in smtlib format.\n3. The arithmetic that are overflow-safe (\"redundant checks\"). Each line\n   corresponds to a subtyping constraint, displayed in the format the format\n   `function: line #: T1 ==\u003e T2` where `T1, T2` are refinement types.\n4. The arithmetic that are overflow-unsafe (\"necessary checks\")\n5. The running time of the inference algorithm. Time spent preprocessing the\n   contract and generating constraints only incurs a small overhead and is\n   therefore not counted in this number.\n\nThe output here says that the ops on lines 8, 10, and 14 are safe (due to the\ncontract invariant, require, and contract invariant, resp.). Note that line 9\nis heuristically detected as a runtime overflow check and is not checked for\nsafety.\n\n### SemiSolid\n\nTo switch to semi-automated mode (e.g., provide contract invariant with other\nannotations inferred), add the `--task check -i '\u003ccontract invariant\u003e'` flags.\n\nThe syntax of the contract invariant is:\n\n* variables, e.g. `totalSupply`, `balances`\n* constants: `true`, `false`, `1`, `1000`, etc.\n* arithmetic, boolean, and relation expressions, e.g. `x + y`, `foo - bar`,\n  `x \u003e= y`, `5 == 10`, `true \u0026\u0026 false`\n* sum of a mapping of uint: `sum(balances)`\n* sum of a field in a mapping of struct: `sum(fld(StructName, fieldName, term))`\n* sum of a nested mapping: `sum(flatten(term))`\n\nFor example:\n\n```bash\n$ solid --solc solc_0.4.26 data/sanity_test.sol --task check -i 'x \u003c= 100'\n\nNow running on SanityTest\nOK\n```\n\nIf we set the contract invariant to `true`, line 14 should no longer be marked\nas safe:\n\n```bash\n$ solid --solc solc_0.4.26 data/sanity_test.sol --task check -i true\n\nNow running on SanityTest\nVIOLATED\nERR safe math: bar: line 14: {v:bool | v == ((x$3 + (cast[uint256](5))) \u003c= 115792089237316195423570985008687907853269984665640564039457584007913129639935)} ==\u003e {v:bool | v == true}\n```\n\nNote: The output of SemiSolid currently does not distinguish between UNSAT and\ntimeout, and all timeouts are marked as unsafe.\n\n## Claims and Evaluation Steps\n\nWe make the following claims in our paper:\n\n1. Solid runs faster than VeriSmart.\n2. Solid achieves a lower false positive rate than VeriSmart. Other metrics such\n   as number of ops marked overflow-safe are also better in Solid.\n3. Contract invariant inference is important for detecting redundant overflow\n   checks in Solidity.\n4. Sum properties (incl. those over nested data structures) are useful for\n   proving overflow safety in Solidity.\n\nTo evaluate these claims, we suggest following the evaluation steps listed\nbelow.\n\n1. Run `bash run_benchmarks.sh` to generate output in the `output/` folder.\n   This may take up to 10 minutes (or more) to run depending on the value of\n   `QUERY_TIMEOUT` in the script (see section below).\n\n   Note that `run_benchmarks.sh` will skip a Solid run on a contract if the\n   corresponding log file already exists.\n2. Each contract will have a corresponding folder in the `output/` folder, e.g.\n   `output/sanity_test.sol/`. In this folder, locate `auto.log` and\n   `semi-true.log`. The former is for AutoSolid results and the latter is\n   SemiSolid results with true as the contract invariant.\n2. For each `output/$NAME/auto.log` file, inspect the running time at the end of\n   the file. Verify claim 1 by checking that the time is significantly faster\n   than the time reported at the bottom of `author-log/$NAME/verismart.log`.\n3. Now, compare each `output/$NAME/auto.log` file to each `verismart.log` file.\n   Verify claim 2 by checking that Solid has comparable or fewer false\n   positives. We've inserted ground truth annotations as comments in the\n   contract source code, where each comment is prefixed by `EVAL: `.\n   \n   This is a labor-intensive step and may require as much as 1 hour to complete.\n4. Using the contract invariants in `output/$NAME/auto.log` (or a stronger one\n   that you can determine yourself), run Solid in SemiSolid mode on each\n   contract. This can be done either by manually invoking `solid` or by using\n   the `semisolid.sh` script. Make sure that the query timeout is set to the\n   same value used in `run_benchmarks.sh` (either as an argument to `solid` or\n   set as a variable at the top of `semisolid.sh`). For example, the following\n   commands will perform this step on `sanity_test.sol` and log the output to\n   `output/sanity_test.sol/semi.log`:\n   \n   ```bash\n   # manually invoking solid\n   (time solid --solc solc_0.4.26 --only-last --query-timeout 10000 --total-timeout 600 data/sanity_test.sol --task check -i 'x \u003c= 100') 2\u003e\u00261 | tee output/sanity_test.sol/semi.log\n   \n   # using semisolid.sh\n   bash ./semisolid.sh data/sanity_test.sol 'x \u003c= 100'\n   ```\n   \n   These commands will overwrite an existing log file, so be careful when running them.\n   \n   Verify claim 3 by comparing the results to `output/$NAME/semi-true.log`.\n\n## Additional information\n\n\u003e Possible differences in the number of operations reported by VeriSmart and Solid\n\nSolid's frontend contains transformation which may add or delete operations:\n\n* Solid may inline internal function calls (up to depth 1). This is currently accomplished by copying the body of the inlined function into the enclosing function, leading to duplicated operations.\n* Solid may partially evaluate expressions involving constants such as `10**8`, effectively removing operations.\n\nThe operations reported by Solid correspond to the `[IO]` category in VeriSmart.\n\n\u003e On running time\n\nMost of the running time is spent invoking the CHC solver. The running time of\nthe tool (and therefore the results) may vary greatly depending on operating\nsystem and CPU. You may need to adjust the `QUERY_TIMEOUT` variable in\n`run_benchmarks.sh`, where the timeout is in milliseconds. We recommend that the\nreviewers experiment with timeouts of `1000`, `5000`, and `10000` (default).\n\nFor comparison, the experiments in the paper were conducted with an AMD Ryzen 9\n5900X CPU and 32GB of RAM with a `QUERY_TIMEOUT` of 10000 ms.\n\n\u003e ```\n\u003e solid: solc: createProcess: runInteractiveProcess: exec: does not exist (No such file or directory)\n\u003e ```\n\nThe image contains two solc binaries named `solc_0.4.26` and `solc_0.5.17`. They\nmust be explicitly passed to solid, e.g.\n\n```\nsolid --solc solc_0.5.17 data/sanity_test.sol\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftechnius%2Fsoltype-artifact-public","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftechnius%2Fsoltype-artifact-public","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftechnius%2Fsoltype-artifact-public/lists"}