https://github.com/scttfrdmn/substrate
Event-sourced AWS emulator for deterministic testing of AI-generated infrastructure code
https://github.com/scttfrdmn/substrate
aws aws-cdk aws-emulator boto3 cloudformation devops event-sourcing go golang infrastructure-as-code integration-testing localstack-alternative mock terraform testing
Last synced: about 7 hours ago
JSON representation
Event-sourced AWS emulator for deterministic testing of AI-generated infrastructure code
- Host: GitHub
- URL: https://github.com/scttfrdmn/substrate
- Owner: scttfrdmn
- License: other
- Created: 2026-03-07T23:22:15.000Z (4 months ago)
- Default Branch: main
- Last Pushed: 2026-06-10T05:37:13.000Z (26 days ago)
- Last Synced: 2026-06-10T06:28:10.959Z (26 days ago)
- Topics: aws, aws-cdk, aws-emulator, boto3, cloudformation, devops, event-sourcing, go, golang, infrastructure-as-code, integration-testing, localstack-alternative, mock, terraform, testing
- Language: Go
- Homepage: https://scttfrdmn.github.io/substrate/
- Size: 1.79 MB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
- Security: SECURITY.md
Awesome Lists containing this project
README
# Substrate
> **The test harness for AI-generated infrastructure.**
> Deterministic. Time-travel debuggable. Cost-visible.
[](https://github.com/scttfrdmn/substrate/actions/workflows/ci.yml)
[](https://goreportcard.com/report/github.com/scttfrdmn/substrate)
[](LICENSE)
## What is Substrate?
Substrate is an **event-sourced AWS emulator** for testing the infrastructure
code that drives AWS — CloudFormation, CDK, Terraform, and any SDK or CLI call —
**deterministically, offline, and with cost visibility**, before you deploy to a
real account.
It models **what is observable through an AWS API call** — request/response
shapes, resource state and how it transitions over a simulated clock, error
codes, and seedable outcomes — *not* what software inside a resource does. It
never runs your user-data, your Lambda code, an inference, or a training job;
those are recorded as intent with a **seedable** result. That boundary is what
makes every run reproducible: API observations can be recorded as events and
replayed identically, whereas a real workload's timing, scheduling, and I/O
cannot.
**Use it two ways:**
- **As a server** — run `substrate`, point any AWS SDK/CLI at `http://localhost:4566`.
This is how most consumers use it.
- **As a Go test harness** — `import ".../emulator"`, spin up an in-process server
or deploy a CloudFormation template directly, no HTTP needed.
## The Problem
```
AI generates infrastructure code → ??? → Deploy to AWS → $$$
^
This is where you find out
```
AI generates infrastructure code *at volume*, inside generate → test → fix loops.
That changes what a test harness has to do — and it's where most AWS emulators
fall down.
## Why Substrate for AI-generated infrastructure
A generate-and-verify loop has needs a human-paced workflow doesn't have:
- **Determinism, because flakes mislead the loop.** A non-deterministic failure is
a *false signal an agent acts on* — wasting fix cycles chasing timing noise.
Substrate records every request as an immutable event over a simulated clock, so
a failing run is real and replays identically. No retry-until-green.
- **Free, fast, offline, so you can run it on every generation.** No real account,
no provisioning latency, no spend, no blast radius. Suitable for tight inner
loops and CI on every candidate the model produces.
- **Cost visibility as a machine-gradeable guardrail.** An AI has no instinct for
bill shock. Substrate prices each operation, so you get a projected dollar figure
*before* deploy — and can gate on it: `betty.Deploy(ctx, tmpl, Intent{MaxCost: 1.0})`
fails the run if generated infra would blow a budget.
- **Seedable failure paths, to verify the error handling the code claims to have.**
Generated code is full of retry/poll/wait logic for capacity errors, throttling,
and terminal states. Substrate lets you seed exactly those outcomes on demand, so
the unhappy paths are actually exercised — not just assumed.
## How it compares
Substrate is a **different tier** from container emulators and real accounts, not a
drop-in replacement. It trades workload-execution fidelity for determinism,
replayability, and cost insight — the fast inner-loop tier.
| | **Substrate** | LocalStack | moto | Real AWS |
|---|:---:|:---:|:---:|:---:|
| Deterministic replay | ✅ | ❌ | partial | ❌ |
| Time-travel debugging | ✅ | ❌ | ❌ | ❌ |
| Cost visibility before deploy | ✅ | ❌ | ❌ | after the bill |
| Seedable failure / capacity / timing paths | ✅ | partial | partial | ❌ |
| Runs your actual workload code | ❌ *(by design)* | ✅ | ❌ | ✅ |
| Language-agnostic (any SDK/CLI over HTTP) | ✅ | ✅ | Python-first | ✅ |
| No account · offline · free | ✅ | ✅ | ✅ | ❌ |
The one ❌ is deliberate: Substrate models **what is observable through an AWS API
call**, not what software inside a resource does (see
[Scope & Philosophy](docs/scope.md)). If you need to execute your Lambda's code or
boot a real container, reach for LocalStack or a real account; if you need to test
*how your infrastructure code drives and reacts to the AWS API* — fast,
deterministically, and on every AI-generated candidate — that's Substrate.
## Core capabilities
### 1. Deterministic reproducibility
Every AWS request is an immutable event over a simulated clock. Same inputs +
same seed = same outputs, every time — no flakes, and a failing run replays
identically for debugging.
```go
session, _ := engine.StartRecording(ctx, "test-lambda-timeout")
runTests() // fails
engine.StopRecording(ctx, session)
// Replay it 1000 times — identical failure every time.
for range 1000 {
results, _ := engine.Replay(ctx, session.StreamID)
}
```
### 2. Time-travel debugging
Step backward through recorded request history and inspect service state at any
point — see exactly where a sequence of API calls diverged from what you expected.
```go
replay := engine.Replay(ctx, "failing-test")
engine.JumpToEvent(ctx, 87) // jump to the failure
engine.StepBackward(ctx) // step back
state, _ := engine.InspectState(ctx, "iam") // see what broke
```
### 3. Cost visibility before deploy
Real AWS pricing tracked per operation. Know your monthly bill before it arrives.
```
Total: $1,247.50/month
S3 PUT: $875.00 (175M ops @ $0.005/1K)
Lambda: $267.50 (10M invocations)
WARNING: High S3 PUT rate — consider batching (save ~99%)
```
### 4. Seedable outcomes (API-surface scope)
Determinism doesn't mean every test sees the same result. Substrate defaults to
the nominal success path; a test **seeds** an alternate outcome — an
`InsufficientInstanceCapacity` on launch, a training job that comes back `Failed`
with a `CapacityError`, a specific query result — and the plugin returns it at
request time, fully reproducibly. The failure, capacity, and timing paths that
are rare or impossible to trigger against real AWS become trivial to test.
## Quick Start (server)
The primary way to use Substrate is as a drop-in AWS endpoint.
### Install
```bash
go install github.com/scttfrdmn/substrate/cmd/substrate@latest
```
Or build from source / run with Docker:
```bash
git clone https://github.com/scttfrdmn/substrate && cd substrate
make build # produces ./bin/substrate
docker run -p 4566:4566 ghcr.io/scttfrdmn/substrate:latest
```
### Run the server
```bash
substrate server
# Listening on :4566
```
Configuration via `substrate.yaml` or environment variables (see
[`substrate.yaml.example`](substrate.yaml.example)).
### Point your AWS client at it
AWS CLI
```bash
aws iam create-user --user-name alice \
--endpoint-url http://localhost:4566 --region us-east-1 --no-sign-request
```
Or set a profile in `~/.aws/config`:
```ini
[profile substrate]
region = us-east-1
endpoint_url = http://localhost:4566
```
Go SDK v2
```go
cfg, _ := config.LoadDefaultConfig(context.TODO(),
config.WithRegion("us-east-1"),
config.WithBaseEndpoint("http://localhost:4566"),
config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider("test", "test", "")),
)
client := iam.NewFromConfig(cfg)
```
Python (boto3)
```python
import boto3
client = boto3.client(
"iam", region_name="us-east-1",
endpoint_url="http://localhost:4566",
aws_access_key_id="test", aws_secret_access_key="test",
)
```
Node.js (AWS SDK v3)
```javascript
import { IAMClient } from "@aws-sdk/client-iam";
const client = new IAMClient({
region: "us-east-1",
endpoint: "http://localhost:4566",
credentials: { accessKeyId: "test", secretAccessKey: "test" },
});
```
## Use as a Go test harness
For Go code, the fastest path is `StartTestServer`, which spins up an in-process
server on a random port and registers a `t.Cleanup` to shut it down.
```go
import "github.com/scttfrdmn/substrate/emulator"
func TestMyInfra(t *testing.T) {
ts := emulator.StartTestServer(t)
cfg, _ := config.LoadDefaultConfig(context.Background(),
config.WithRegion("us-east-1"),
config.WithBaseEndpoint(ts.URL),
config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider("test", "test", "")),
)
// Use cfg with any AWS SDK v2 client...
}
```
Or deploy a CloudFormation template and validate it entirely in-process, no HTTP
server required, via the **Betty** client:
```go
import "github.com/scttfrdmn/substrate/emulator"
store := emulator.NewEventStore(emulator.EventStoreConfig{Enabled: true, Backend: "memory"})
state := emulator.NewMemoryStateManager()
tc := emulator.NewTimeController(time.Now())
registry := emulator.NewPluginRegistry()
betty := emulator.NewBettyClient(registry, store, state, tc, logger)
result, _ := betty.Deploy(ctx, cfnTemplate, emulator.Intent{MaxCost: 1.0})
session, _ := betty.StartRecording(ctx, "my-test")
// ... run operations against the emulator ...
report, _ := betty.StopRecording(ctx, session)
fmt.Printf("status=%s cost=$%.4f\n", report.PassFail, report.Cost.Total)
```
> The Go import path is `github.com/scttfrdmn/substrate/emulator`. Installing the
> CLI (`.../cmd/substrate@latest`) is unaffected.
See [`examples/betty_workflow/main.go`](examples/betty_workflow/main.go) for a
complete runnable example.
## Supported services
Substrate ships **63 built-in service plugins** spanning compute, storage,
networking, databases, messaging, analytics, ML, security, and management:
- **Compute & containers** — EC2, Lambda, ECS, ECR, Batch, EKS-adjacent
- **Storage & data** — S3, EFS, FSx, DynamoDB, RDS, Redshift, ElastiCache, Timestream
- **Networking** — VPC/EC2, ELBv2, Route 53, CloudFront, API Gateway (REST & HTTP)
- **Messaging & streaming** — SQS, SNS, EventBridge, Kinesis, Firehose, MSK
- **ML & analytics** — SageMaker, Bedrock Runtime, Athena, Glue, EMR Serverless, OpenSearch, QuickSight, HealthOmics
- **Security & identity** — IAM, STS, KMS, Secrets Manager, ACM, Cognito, SSO, WAFv2, RAM
- **Management & cost** — CloudWatch (+ Logs), CloudTrail, Organizations, Budgets, Cost Explorer, Service Quotas, SSM, Health, the CodeSuite, Step Functions, Backup, Transfer
Many integrate with **Betty** for CloudFormation deployment. See the
[Service Reference](docs/services.md) for the authoritative, per-operation list.
### Known limitations
- **Cross-service IAM enforcement** — policies are evaluated for IAM and STS;
per-operation enforcement for other services is partial.
- **Persistence** — in-memory by default; SQLite available via
`EventStoreConfig{Backend: "sqlite"}`.
- **Authentication** — SigV4 verification is opt-in (off by default for testing
ease); enable with `ServerOptions.VerifySignatures = true`.
- **Workload internals are out of scope by design** — Substrate models the API
surface, not what runs inside a resource (see *What is Substrate?*).
## Status
Current release: **v0.68.0**. See [Releases](https://github.com/scttfrdmn/substrate/releases)
and [CHANGELOG.md](CHANGELOG.md) for full history.
## Documentation
📖 **Full documentation: [scttfrdmn.github.io/substrate](https://scttfrdmn.github.io/substrate/)**
- **[Getting Started](docs/getting-started.md)** — install, first test, 15-minute tutorial
- **[Service Reference](docs/services.md)** — all 63 plugins with operation lists
- **[Testing Guide](docs/testing-guide.md)** — `StartTestServer`, recording/replay, cost assertions
- **[Endpoint Configuration](docs/endpoint-configuration.md)** — SDK and tool configuration
## Development
```bash
make test # run tests with the race detector
make lint # golangci-lint
make coverage # coverage report
make build # build the substrate binary
make e2e # end-to-end tests
```
Requirements: Go 1.26+, [golangci-lint](https://golangci-lint.run/).
## Contributing
Issues and pull requests welcome. All work is tracked in
[GitHub Issues](https://github.com/scttfrdmn/substrate/issues) and organised into
[Milestones](https://github.com/scttfrdmn/substrate/milestones). `main` is
protected — changes land via pull request.
## License
Apache 2.0 — see [LICENSE](LICENSE).