https://github.com/patrick96/smtransform
Find soundness bugs in SMT solvers through equivalent transformations
https://github.com/patrick96/smtransform
smt-lib smt-solving
Last synced: 6 months ago
JSON representation
Find soundness bugs in SMT solvers through equivalent transformations
- Host: GitHub
- URL: https://github.com/patrick96/smtransform
- Owner: patrick96
- License: mit
- Created: 2022-04-13T12:50:50.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2022-06-09T10:25:20.000Z (over 3 years ago)
- Last Synced: 2025-03-27T14:06:33.814Z (6 months ago)
- Topics: smt-lib, smt-solving
- Language: Rust
- Homepage:
- Size: 314 KB
- Stars: 6
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# SMTransform
*SMTransform* is framework with the goal to find unsoundness bugs in SMT
solvers through fuzzing.
From a satisfiable seed formula, *SMTransform* incrementally generates new
formulas which are satisfiable by construction.
## Installation
### Requirements
* [Cargo](https://github.com/rust-lang/cargo/)
* Python 3.9+
The project is used and developed on Linux, but it should work on any platform
supported by rust and python.
### Building
Compiling the project is as easy as running
```bash
cargo build --release
```
The `smtransform` executable will then be placed in `target/release/smtransform`.
## Usage
```
USAGE:
smtransform [OPTIONS]
ARGS:
File containing seed formula in the SMT-LIB2 format
OPTIONS:
-h, --help Print help information
--json Print generated formulas wrapped in JSON object
--print-original Also print the seed formula
--rounds How many formulas to generate [default: 1]
--seed Seed for the PRNG [default: 0]
-V, --version Print version information
```
Sample usage:
```
$ cat in.smt2
(set-info :status sat)
(set-logic QF_NIA)
(declare-fun x () Int)
(declare-fun y () Int)
(assert (= x 3))
(assert (= y 9))
(check-sat)
(exit)
$ ./target/release/smtransform in.smt2
(set-logic QF_NIA)
(set-info :status sat)
(declare-fun x () Int)
(declare-fun y () Int)
(declare-fun v0 () Int)
(assert (= (- v0 y) 3))
(assert (= (- v0 x) 9))
(check-sat)
(exit)
```
A large collection of seed formulas is available in the [SMT-LIB benchmark
repository](https://clc-gitlab.cs.uiowa.edu:2443/SMT-LIB-benchmarks).
### Helper Scripts
#### `scripts/runner.py`
Pipe the JSON output from the `smtransform` executable into the runner script:
```bash
./target/release/smtransform --json in.smt2 | ./scripts/runner.py -- z3 /dev/stdin
```
This will run each generated formula through the given SMT solver (the solver
command should accept a formula on stdin) and report each failure.
### `scripts/wrapper.py`
Runs the generator for multiple seed formulas, passes each formula to
`runner.py`, and collects failures in the given output directory.
```
$ ./scripts/wrapper.py \
--seeds seeds \
--out out \
--gen target/release/smtransform \
--iterations 20 \
--rounds 10 \
-- z3 /dev/stdin
Using seed seeds/20170427-VeryMax_ITS_From_T2__ex36.t2_fixed__p23162_safety_0.smt2 20/20: T..........
Using seed seeds/20170427-VeryMax_ITS_From_T2__ex36.t2_fixed__p23504_safety_0.smt2 20/20: TTTTT......
Using seed seeds/20170427-VeryMax_ITS_From_T2__s1.t2_fixed__p15649_safety_0.smt2 20/20: ...........
Using seed seeds/20170427-VeryMax_SAT14_279.smt2 20/20: T..........
Using seed seeds/AProVE_aproveSMT1179149253451050796.smt2 20/20: ..T......T.
Using seed seeds/AProVE_aproveSMT2322179511678559502.smt2 20/20: ....TTT.TTT
Using seed seeds/AProVE_aproveSMT2821114236865559656.smt2 20/20: T......TT..
Using seed seeds/AProVE_aproveSMT4248959283647504950.smt2 20/20: ..........T
Using seed seeds/AProVE_aproveSMT4556648541890562734.smt2 20/20: .T.........
Using seed seeds/AProVE_aproveSMT5479381539249843197.smt2 20/20: ......TTTTT
Success: 1611
Failure: 3
Timeout: 586
Signal: 0
Unknown: 0
Unsound: 0
Runs per iteration (approx): 11
Runs per seed (approx): 220
Seeds: 10
Total runs: 2200
```
## Limitations / TODOs
* There is no type checker, so we don't really know the type of a term unless it is a variable with known type.
* The `ConstantReplacement` transformation currently assumes `NUMERAL` is
of sort `Int`, which would be unsound if the numeral was actually used as a
`Real`.
* The `AssertExpandBinary` transformations assumes that function names from
the standard SMT-LIB theories are not overloaded (e.g. `and`, `>=`, etc.)
because it needs to assume the function signature.