https://github.com/fossable/attest
Dead simple test framework for the age of AI
https://github.com/fossable/attest
shell shell-scripting test-automation testing testing-tools
Last synced: 5 days ago
JSON representation
Dead simple test framework for the age of AI
- Host: GitHub
- URL: https://github.com/fossable/attest
- Owner: fossable
- License: unlicense
- Created: 2026-04-23T02:58:27.000Z (2 months ago)
- Default Branch: master
- Last Pushed: 2026-06-03T02:52:43.000Z (23 days ago)
- Last Synced: 2026-06-03T04:26:10.234Z (23 days ago)
- Topics: shell, shell-scripting, test-automation, testing, testing-tools
- Language: Rust
- Homepage:
- Size: 848 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Agents: AGENTS.md
Awesome Lists containing this project
README
> perfection is finally attained not when there is no longer anything to add,
> but when there is no longer anything to take away
>
> Terre des Hommes (1939) - Antoine de Saint Exupéry
**attest** is a simple and modern test framework for CLI programs. There is no
exotic test syntax to remember, assertion API, plugins, or hidden lifecycle
methods to know about. Tests are just regular shell functions where every
statement is an assertion.
We already have all of the tools we need to write tests in the shell:
- Shell functions neatly organize tests into runnable units
- Need to compare text? `[` and `[[` have been around for decades
- Need to compare JSON? `jq -c` has you covered.
- Need some test setup/cleanup? Idiomatic with helper functions and traps.
By keeping the framework lightweight, tests are easy to write and quick to
understand, leading to an overall more effective testing experience.
## Writing tests
Here's an illustrative example of a test for the `md5sum` command:
```sh
## Test the md5sum command with known input/output
testHello() {
result=$(echo hello | md5sum) # Test fails if nonzero exit
[ "${result}" = "b1946ac92492d2347c6235b4d2611184 -" ] # Test fails if output changes
}
```
It looks like an ordinary shell script because it **is** an ordinary shell
script. You could even source it into your shell and run it directly if you
wanted to. Don't try that with Bats :).
There are only three implicit pieces of knowledge that you need for writing
tests:
- All test functions are named starting with `test`
- If any command in your function exits nonzero, the whole test fails
- Each test runs in a separate temporary directory
### Inline tests
If you're testing something that's itself a shell script, you can also include
your tests inline with the script.
For example:
```sh
#!/usr/bin/env bash
## Inline tests can be placed anywhere in the script. You can use $0 to call
## the script we're embedded in.
testGoodInput() {
result=$($0 1 2)
[ ${result} -eq 3 ]
}
testNoInput() {
! $0
}
testBadInput() {
! $0 1 1.2
}
testTooManyArgs() {
! $0 1 2 3
}
# Here is the actual implementation of the script. It's not important for our
# purposes; I just prompted Claude for the most complicated way to add two
# numbers. The model calls it "enterprise grade" :)
python3 -c "
import subprocess, json, sys, os, tempfile, re
def validate(x):
result = subprocess.run(
['bash', '-c', f'printf \"%d\" \"{x}\" 2>/dev/null || exit 1'],
capture_output=True, text=True
)
if result.returncode != 0:
raise ValueError(f'not an integer: {x}')
return x
def add(a, b):
validate(a)
validate(b)
with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f:
json.dump({'operands': [a, b], 'operation': 'addition'}, f)
fname = f.name
with open(fname) as f:
payload = json.load(f)
os.unlink(fname)
result = subprocess.run(
['awk', '-v', f'a={payload[\"operands\"][0]}', '-v', f'b={payload[\"operands\"][1]}',
'BEGIN { print a + b }'],
capture_output=True, text=True
)
return result.stdout.strip()
if len(sys.argv) != 3:
sys.exit(1)
try:
print(add(sys.argv[1], sys.argv[2]))
except ValueError:
sys.exit(1)
" "$1" "$2"
```
Inline tests can be nice because the closer the tests are to the code that's
being tested, the more likely they are maintained over time.
### Generating tests with AI
Tests are also easy for AIs to write. Use `attest skill` to print the a skill
which is pre-tuned to produce good tests. Here's an example prompt:
> Write attest-style tests for the `tac` command.
Here's the output from Haiku 4.5:
```sh
#!/usr/bin/env bash
## Basic functionality - reverse lines from a file
testBasicFile() {
echo -e "line1\nline2\nline3" > input.txt
result=$(tac input.txt)
[ "$result" = "$(echo -e "line3\nline2\nline1")" ]
}
## Reverse lines from stdin
testStdin() {
result=$(echo -e "first\nsecond\nthird" | tac)
[ "$result" = "$(echo -e "third\nsecond\nfirst")" ]
}
## Single line should remain unchanged
testSingleLine() {
echo "only line" > input.txt
result=$(tac input.txt)
[ "$result" = "only line" ]
}
## Empty file produces no output
testEmptyFile() {
touch empty.txt
result=$(tac empty.txt)
[ -z "$result" ]
}
## Multiple files are concatenated and reversed
testMultipleFiles() {
echo -e "a\nb" > file1.txt
echo -e "c\nd" > file2.txt
result=$(tac file1.txt file2.txt)
[ "$result" = "$(echo -e "d\nc\nb\na")" ]
}
## Non-existent file produces error
testNonExistentFile() {
! tac nonexistent.txt 2>/dev/null
}
## Trailing newline handling
testTrailingNewline() {
printf "line1\nline2\nline3" > input.txt
result=$(tac input.txt)
[ "$result" = "$(echo -e "line3\nline2\nline1")" ]
}
## Help flag works
testHelp() {
tac --help || tac -h
}
## Version flag works
testVersion() {
tac --version || tac -v
}
## Lines with special characters
testSpecialCharacters() {
echo -e "hello\$world\nfoo@bar\n#comment" > input.txt
result=$(tac input.txt)
echo "$result" | grep -q "^#comment"
echo "$result" | grep -q "foo@bar"
}
## File with only whitespace lines
testWhitespaceLines() {
printf "line1\n \nline3\n" > input.txt
result=$(tac input.txt)
[ "$(echo "$result" | wc -l)" -eq 3 ]
}
## Very long lines are handled correctly
testLongLines() {
long_line=$(printf 'a%.0s' {1..1000})
echo -e "$long_line\nshort" > input.txt
result=$(tac input.txt)
echo "$result" | grep -q "^short$"
}
## Reads from multiple sources correctly
testMixedInput() {
echo "file line" > file.txt
echo "stdin line" | tac - file.txt | head -1 | grep -q "file line"
}
```
AI can generate tests all day, so the important thing is how easy it is for a
human to quickly understand and assess the quality of AI-produced tests.
## Running tests
Now that we have some tests, AI-generated or not, it's time for the good part.
```sh
# Just run the tests in one file
attest example.test
# Run all tests in this directory
attest .
# Tests run in parallel by default; use --parallel to limit concurrency
attest --parallel 1 .
```
Every test runs in a temporary _context directory_ that collects logs and
temporary files created by the test.
### Containerized tests
If your application requires some dependencies in a Docker container, you can
run `attest` in a container with this recipe:
```sh
docker run --rm -v $(which attest):/bin/attest -v $(pwd):/tests attest /tests
```
### Fuzz testing
If your application spawns subprocesses, `attest` can randomly inflate the
timing of those subprocesses at random:
```sh
attest --fuzz examples/race_condition.test
```
This works by choosing a subprocess at random and sending `SIGSTOP` followed by
`SIGCONT`. This option also works nicely with `--repeat`.
Example
Without `--fuzz`, you might not realize there's a nasty race condition hiding in
this file:
```
❯ attest --parallel 1 --repeat 10 examples/race_condition.test
PASS testGrepQ#1 (1.06s)
cpu=7.8ms+4.8ms mem=2.8MiB pids=5
PASS testGrepQ#2 (1.07s)
cpu=6.2ms+6.2ms mem=3.1MiB pids=5
PASS testGrepQ#3 (1.07s)
cpu=6.2ms+6.2ms mem=2.6MiB pids=5
PASS testGrepQ#4 (1.07s)
cpu=6.2ms+6.2ms mem=3.4MiB pids=5
PASS testGrepQ#5 (1.07s)
cpu=6.3ms+6.3ms mem=3.4MiB pids=5
PASS testGrepQ#6 (1.06s)
cpu=8.7ms+3.7ms mem=3.4MiB pids=5
PASS testGrepQ#7 (1.07s)
cpu=5.9ms+6.9ms mem=3.0MiB pids=5
PASS testGrepQ#8 (1.07s)
cpu=6.3ms+6.3ms mem=3.1MiB pids=5
PASS testGrepQ#9 (1.06s)
cpu=6.1ms+6.1ms mem=3.1MiB pids=5
PASS testGrepQ#10 (1.07s)
cpu=6.2ms+6.2ms mem=2.8MiB pids=5
Results: 10 passed, 10 total
Time: 10.69s
```
Now let's add some fuzziness to the timing:
```
❯ attest --parallel 1 --fuzz 0.9 --repeat 10 examples/race_condition.test 10s
PASS testGrepQ#1 (3.47s)
cpu=5.5ms+7.3ms mem=2.8MiB pids=5
FAIL testGrepQ#2 (4.09s)
cpu=6.7ms+6.0ms mem=3.2MiB pids=5
FAIL testGrepQ#3 (5.10s)
cpu=4.7ms+7.8ms mem=2.9MiB pids=5
PASS testGrepQ#4 (3.59s)
cpu=6.3ms+6.3ms mem=3.5MiB pids=5
FAIL testGrepQ#5 (2.39s)
cpu=5.8ms+6.8ms mem=3.1MiB pids=5
PASS testGrepQ#6 (6.51s)
cpu=5.3ms+7.4ms mem=3.1MiB pids=5
FAIL testGrepQ#7 (3.08s)
cpu=7.1ms+5.7ms mem=3.1MiB pids=5
FAIL testGrepQ#8 (3.34s)
cpu=4.3ms+8.1ms mem=3.4MiB pids=5
PASS testGrepQ#9 (2.68s)
cpu=6.3ms+6.3ms mem=3.1MiB pids=5
FAIL testGrepQ#10 (2.08s)
cpu=6.2ms+6.2ms mem=2.6MiB pids=5
Results: 4 passed, 6 failed, 10 total
Time: 36.35s
```
We were able to shake out the race condition by adding random delays in the
test. The `grep -q` example above is obviously contrived, but imagine you were
checking for firewall rules with `iptables | grep -q`.
You'll also notice the test took over 3 times longer. You can adjust how
aggressive the fuzzer is by passing a higher number to `--fuzz`.
## Debugging tests
When a test fails, you can obtain the context directory:
```sh
attest . --save-context ./results
```
This directory contains everything: the test's xtrace, stdout, any files created
by the tests, etc.
You can also just view the xtrace output with the `--xtrace` flag:
## Installation
Crates.io
#### Install from crates.io
```sh
cargo install attest
```