Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/technius/soltype-artifact-public


https://github.com/technius/soltype-artifact-public

Last synced: 10 months ago
JSON representation

Awesome Lists containing this project

README 

          
# SolType artifact



To facilitate artifact evaluation, we have packaged our binaries as a docker

image, with the supporting benchmarks and logs stored in this directory. You can

retrieve the docker image using the following command:



```bash

docker pull technius/soltype-artifact:latest

```



We describe some of the supporting files below:

* `run_benchmarks.sh`: generates output from Solid on the representative

  benchmark files.

* `solc_vers.csv`: contains solidity compiler version information

* `cinv.csv`: lists the contract invariant associated with each contract

* `author-logs/`: contains log files that were generated by the authors

* `data/`: contains the representative benchmark files



Due to the large number of benchmark files in our evaluation, we think it will

be too time consuming to verify the claims for all 120 benchmarks. Thus, we have

selected 5 representative benchmarks for which be believe the claims can be

verified within a reasonable amount of time. We are happy to provide more

benchmarks upon request.



## Setup and Usage



In a shell, navigate to this directory.



Then, launch the prepackaged environment using docker:



```bash

sudo docker run --rm -it -v "$PWD":/opt/eval technius/soltype-artifact:latest

```



This will mount the artifact data folder into the image and then place you

into a bash shell. Make sure you are able to locate `data/sanity_test.sol`

using the `ls` command.



Now, check that the `solid` command produces help output:



```bash

solid -h

```



In general, AutoSolid can be run using the format:



```bash

solid --solc  --only-last 

```



This will run Solid on the last (concrete) contract contained in the provided

`.sol` file.



SemiSolid can be run using:



```bash

solid --solc  --only-last  --task check -i 

```



Two versions of `solc` are available in the image: `solc_0.4.26` and `solc_0.5.17`.



### AutoSolid



To ensure that everything works correctly, try the sanity test:



```bash

$ solid --solc solc_0.4.26 data/sanity_test.sol



Now running on SanityTest



Bootstrapping...

Trying templates...

[]

end templates

Iteration 0

safe: [8,3,2]

List [Atom "not",List [Atom ">=",Atom "x!0",Atom "101"]]

Total math ops: 3

Provably safe math ops: 3

Inferred contract invariant: (not (>= x 101 ) )

The following safe math checks are redundant:

bar: line 14: {v:bool | v == ((x$3 + (cast[uint256](5))) <= 115792089237316195423570985008687907853269984665640564039457584007913129639935)} ==> {v:bool | v == true}

foo: line 10: {v:bool | v == ((x$2 + a$0) <= 115792089237316195423570985008687907853269984665640564039457584007913129639935)} ==> {v:bool | v == true}

foo: line 8: {v:bool | v == ((x$1 + 1) <= 115792089237316195423570985008687907853269984665640564039457584007913129639935)} ==> {v:bool | v == true}

The following safe math checks are necessary:

Solving time: 0.030567557s

```



This will run Solid in automated mode. The output is divided into the following

sections:



1. "Live" output tracking when a query is UNSAT or times out. Queries that time

   out will be retried up to 3 times.

2. Once the algorithm finishes, the inferred contract invariant will be

   displayed (`Inferred contract invariant: ...`). As Solid is still a

   prototype, the invariant is displayed in smtlib format.

3. The arithmetic that are overflow-safe ("redundant checks"). Each line

   corresponds to a subtyping constraint, displayed in the format the format

   `function: line #: T1 ==> T2` where `T1, T2` are refinement types.

4. The arithmetic that are overflow-unsafe ("necessary checks")

5. The running time of the inference algorithm. Time spent preprocessing the

   contract and generating constraints only incurs a small overhead and is

   therefore not counted in this number.



The output here says that the ops on lines 8, 10, and 14 are safe (due to the

contract invariant, require, and contract invariant, resp.). Note that line 9

is heuristically detected as a runtime overflow check and is not checked for

safety.



### SemiSolid



To switch to semi-automated mode (e.g., provide contract invariant with other

annotations inferred), add the `--task check -i ''` flags.



The syntax of the contract invariant is:



* variables, e.g. `totalSupply`, `balances`

* constants: `true`, `false`, `1`, `1000`, etc.

* arithmetic, boolean, and relation expressions, e.g. `x + y`, `foo - bar`,

  `x >= y`, `5 == 10`, `true && false`

* sum of a mapping of uint: `sum(balances)`

* sum of a field in a mapping of struct: `sum(fld(StructName, fieldName, term))`

* sum of a nested mapping: `sum(flatten(term))`



For example:



```bash

$ solid --solc solc_0.4.26 data/sanity_test.sol --task check -i 'x <= 100'



Now running on SanityTest

OK

```



If we set the contract invariant to `true`, line 14 should no longer be marked

as safe:



```bash

$ solid --solc solc_0.4.26 data/sanity_test.sol --task check -i true



Now running on SanityTest

VIOLATED

ERR safe math: bar: line 14: {v:bool | v == ((x$3 + (cast[uint256](5))) <= 115792089237316195423570985008687907853269984665640564039457584007913129639935)} ==> {v:bool | v == true}

```



Note: The output of SemiSolid currently does not distinguish between UNSAT and

timeout, and all timeouts are marked as unsafe.



## Claims and Evaluation Steps



We make the following claims in our paper:



1. Solid runs faster than VeriSmart.

2. Solid achieves a lower false positive rate than VeriSmart. Other metrics such

   as number of ops marked overflow-safe are also better in Solid.

3. Contract invariant inference is important for detecting redundant overflow

   checks in Solidity.

4. Sum properties (incl. those over nested data structures) are useful for

   proving overflow safety in Solidity.



To evaluate these claims, we suggest following the evaluation steps listed

below.



1. Run `bash run_benchmarks.sh` to generate output in the `output/` folder.

   This may take up to 10 minutes (or more) to run depending on the value of

   `QUERY_TIMEOUT` in the script (see section below).



   Note that `run_benchmarks.sh` will skip a Solid run on a contract if the

   corresponding log file already exists.

2. Each contract will have a corresponding folder in the `output/` folder, e.g.

   `output/sanity_test.sol/`. In this folder, locate `auto.log` and

   `semi-true.log`. The former is for AutoSolid results and the latter is

   SemiSolid results with true as the contract invariant.

2. For each `output/$NAME/auto.log` file, inspect the running time at the end of

   the file. Verify claim 1 by checking that the time is significantly faster

   than the time reported at the bottom of `author-log/$NAME/verismart.log`.

3. Now, compare each `output/$NAME/auto.log` file to each `verismart.log` file.

   Verify claim 2 by checking that Solid has comparable or fewer false

   positives. We've inserted ground truth annotations as comments in the

   contract source code, where each comment is prefixed by `EVAL: `.

   

   This is a labor-intensive step and may require as much as 1 hour to complete.

4. Using the contract invariants in `output/$NAME/auto.log` (or a stronger one

   that you can determine yourself), run Solid in SemiSolid mode on each

   contract. This can be done either by manually invoking `solid` or by using

   the `semisolid.sh` script. Make sure that the query timeout is set to the

   same value used in `run_benchmarks.sh` (either as an argument to `solid` or

   set as a variable at the top of `semisolid.sh`). For example, the following

   commands will perform this step on `sanity_test.sol` and log the output to

   `output/sanity_test.sol/semi.log`:

   

   ```bash

   # manually invoking solid

   (time solid --solc solc_0.4.26 --only-last --query-timeout 10000 --total-timeout 600 data/sanity_test.sol --task check -i 'x <= 100') 2>&1 | tee output/sanity_test.sol/semi.log

   

   # using semisolid.sh

   bash ./semisolid.sh data/sanity_test.sol 'x <= 100'

   ```

   

   These commands will overwrite an existing log file, so be careful when running them.

   

   Verify claim 3 by comparing the results to `output/$NAME/semi-true.log`.



## Additional information



> Possible differences in the number of operations reported by VeriSmart and Solid



Solid's frontend contains transformation which may add or delete operations:



* Solid may inline internal function calls (up to depth 1). This is currently accomplished by copying the body of the inlined function into the enclosing function, leading to duplicated operations.

* Solid may partially evaluate expressions involving constants such as `10**8`, effectively removing operations.



The operations reported by Solid correspond to the `[IO]` category in VeriSmart.



> On running time



Most of the running time is spent invoking the CHC solver. The running time of

the tool (and therefore the results) may vary greatly depending on operating

system and CPU. You may need to adjust the `QUERY_TIMEOUT` variable in

`run_benchmarks.sh`, where the timeout is in milliseconds. We recommend that the

reviewers experiment with timeouts of `1000`, `5000`, and `10000` (default).



For comparison, the experiments in the paper were conducted with an AMD Ryzen 9

5900X CPU and 32GB of RAM with a `QUERY_TIMEOUT` of 10000 ms.



> ```

> solid: solc: createProcess: runInteractiveProcess: exec: does not exist (No such file or directory)

> ```



The image contains two solc binaries named `solc_0.4.26` and `solc_0.5.17`. They

must be explicitly passed to solid, e.g.



```

solid --solc solc_0.5.17 data/sanity_test.sol

```