Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/say-5/scanguard


https://github.com/say-5/scanguard

go hardware-in-the-loop instrumentation medical-imaging python scpi test-automation

Last synced: about 2 months ago
JSON representation

Awesome Lists containing this project

README 

          
# scanguard



Instrument test and validation platform for a research microPET scanner.



scanguard automates the pre-scan verification and scan-sequence orchestration

for a small-animal PET scanner used to image rodent brains. It replaces a

manual checklist and a flaky scan-sequence script with three pieces:



1. Automated preflight checks that talk to the scanner over a TCP/IP

   instrument protocol (SCPI) before any scan begins.

2. A scan-sequence orchestrator that drives the motorized bed with setup-state

   verification and wait-and-confirm execution.

3. Resilient instrument transports with stable device addressing.



Everything runs against simulated instruments, so the whole system runs with no

hardware attached. The v1.0.0 release turns the preflight piece into a complete

declarative checklist engine, v2.0.0 builds out the scan-sequence orchestrator

with setup-state verification and wait-and-confirm execution, v3.0.0 makes the

instrument transport resilient and addresses each device by its serial, v4.0.0

turns the scripts into a platform: a run-history store, a REST API and Prometheus

metrics, and a web dashboard that read the reports and run logs the earlier

releases produce, and v5.0.0 hardens the platform for production with

config-driven instrument profiles, a fault-injection chaos suite, SCPI parser

fuzzing, hot-path benchmarks with a regression gate, and operability work

(health and readiness, graceful shutdown, structured logging, and input

validation).



## Components



| Path         | Language | What it does                                                        |

|--------------|----------|---------------------------------------------------------------------|

| `sim/`       | Go       | Simulated SCPI scanner and bed controller over TCP. Test backbone.  |

| `core/`      | Go       | SCPI transport, bed and detector clients, the orchestrator, the run-history store, and the REST API daemon. |

| `preflight/` | Python   | Declarative preflight checklist engine: four check types, three report formats. |

| `web/`       | TypeScript | React and Vite dashboard reading the REST API: preflight, calibration countdown, runs, devices, and trends. |

| `e2e/`       | Go       | End-to-end failure-mode proof: drives the sims and orchestrator as real processes. |

| `chaos/`     | Go       | Fault-injection suite: drives the sims and orchestrator and asserts safe failure or recovery. |

| `instrument/`| Go       | Config-driven instrument profiles and the two shipped profiles. |

| `bench/`     | -        | Benchmark baseline and the regression-gate documentation. |

| `docs/`      | Markdown | Architecture, glossary, operations, and the contribution workflow.  |



## How this differs



The simulators are first-class. Every later release is tested against the same

fake instruments, so the test backbone never depends on lab hardware being

free. The orchestrator and preflight engine speak the same SCPI text protocol

to the simulators that they would speak to the real scanner.



## Preflight checklist



The preflight engine runs the morning scanner-verification checklist. The

checklist is declarative: the checks, their ranges, and their thresholds live

in a JSON config (see `preflight/configs/default.json`), so adding a detector

channel or changing a limit is a config edit, not a code change. The engine

runs every check on each pass and never stops at the first failure, so the

report shows every problem at once and only the final line says whether it is

safe to proceed.



Four check types are supported:



- **Detector channel range**: queries each configured channel over SCPI

  (`MEAS:VOLT? CHAN1`, `CHAN2`, ...) and confirms the voltage is within range.

- **Calibration file**: reads a JSON calibration file's `sensor_gain`,

  `offset_correction`, and `last_calibrated` date, checks the gain and offset

  ranges, and flags a calibration older than 30 days with a distinct stale

  status.

- **Device self test**: sends `*TST?` and flags any non-zero code.

- **Disk space**: confirms the scan-output drive has enough free space.



Run it with `python -m preflight`:



```sh

python -m preflight --config preflight/configs/default.json \

  --scanner 127.0.0.1:5025 \

  --json report.json \

  --junit report.xml

```



The exit code is non-zero if any check fails. There are three output formats:

a human PASS/FAIL summary on stdout, a structured JSON report (`--json`), and a

JUnit XML report (`--junit`) that CI can ingest as a test report.



A passing run looks like this:



```

[ PASS ] Detector Channel 1: 4.97V (range 4.8-5.2V)

[ PASS ] Detector Channel 2: 5.01V (range 4.8-5.2V)

[ PASS ] Device Self Test: self test passed (code 0)

[ PASS ] Calibration File: gain 1.02, offset 0.05, 14 days old

[ PASS ] Disk Space: 6.3GB free, need 1GB

5/5 checks passed

preflight: PASS (proceed)

```



A run with problems reports each one before the verdict:



```

[ PASS ] Detector Channel 1: 4.97V (range 4.8-5.2V)

[ FAIL ] Detector Channel 2: 5.01V (range 2.9-3.1V)

[ FAIL ] Device Self Test: self test failed (code 4)

[ FAIL ] Calibration File: calibration 144 days old, max 30 days

[ FAIL ] Disk Space: 6.1GB free, need 10GB

1/5 checks passed

preflight: FAIL (do not proceed)

```



## Scan-sequence orchestrator



The orchestrator drives the motorized bed through a list of positions. The old

scripts fired commands and moved on at once, so a capture could read the bed

mid-travel and a run could start from whatever state a previous session left

behind. Those runs failed for setup reasons, not real faults, and operators just

reran them. The orchestrator removes that class of false failure without hiding

genuine faults.



It runs in three phases:



1. **Setup verification** runs before any scan step. It reads the bed position

   and, if the bed is not home, sends `MOVE HOME` and polls until it confirms

   the home position (bounded by a timeout). It lists `.dat` files in the output

   directory and aborts, naming the file, if a previous session still holds one

   open. It confirms the detector reports ready. A scan begins only when all

   three pass.

2. **Execution** is wait-and-confirm. For each step it sends `MOVE `, polls

   the bed position until it is within 0.1mm of the target, then captures and

   waits for the capture to complete. Nothing is recorded until the bed is

   actually there.

3. **Logging** writes a structured JSON run file: a run id and timestamp, each

   step with its target position, confirmed position, data points, average

   reading, and timestamp, and a final PASS or FAIL with the reason.



Run it with `scanseq` against the simulators:



```sh

make sim &                                  # simulated scanner on :5025

go run ./sim/cmd/bed -addr 127.0.0.1:5026 & # simulated bed

go run ./core/cmd/scanseq \

  -scanner 127.0.0.1:5025 \

  -bed 127.0.0.1:5026 \

  -positions 10,20,30 \

  -outdir ./out

```



The bed simulator can reproduce the documented failure modes for testing:

`-start 47` leaves the bed off-home, `-stall 5` makes every move stop 5mm short

of its target, and the scanner's `-notready` flag makes the detector report not

ready. The `e2e` package exercises each of these against the orchestrator.



## Resilient transport and device addressing



The shared workstations had three recurring transport problems. v3.0.0 makes the

platform robust to all three, and proves each against the simulators.



- **Timeout and retry.** The transport timeout was hardcoded at 500ms while the

  instruments answer in 600-850ms, so a healthy device read as a failure. The

  per-request timeout is now configurable with a 2000ms default, and a transient

  failure is retried up to three times with doubling backoff before it counts as

  real. The timeout stays bounded on purpose: a very long one would make every

  genuine failure take that long to detect. The scanner's `-latency` flag drives

  the boundary, and the tests show a 700ms device passing at 2000ms and failing

  at 500ms.

- **Stable addressing.** Port names are handed out by connection order and are

  not stable, but every device has a factory serial that never changes. The

  registry (`core/internal/registry`) binds each serial to a permanent logical

  name and resolves a name to whatever address currently carries that serial, so

  a reorder does not break addressing. The scanner's `-serial` flag gives two

  sims distinct identities for the reorder test.

- **udev rules.** The `udevgen` tool is the ops-side mirror: it turns the lab

  device list into a Linux udev rules file pinning each serial to a fixed

  `/dev/scanguard/` symlink. See `docs/udev.md` and the sample in

  `docs/examples/`.



  ```sh

  go run ./core/cmd/udevgen -devices docs/examples/devices.sample.json

  ```



- **Health and reconnect.** A health-check probe and reconnect let a run recover

  from a mid-session link drop instead of aborting. The scanner's `-dropafter`

  flag models the disconnect.



## Observability platform



v4.0.0 adds a run-history store, a REST API with Prometheus metrics, and a web

dashboard, all reading the reports and run logs the earlier releases produce.



The API daemon (`scanguardd`) keeps an embedded SQLite store and serves:



- `GET /api/preflight/latest`, `GET /api/preflight/history`

- `GET /api/runs`, `GET /api/runs/{id}`

- `GET /api/devices`, `GET /api/calibration`, `GET /api/trends`

- `POST /api/preflight/reports`, `POST /api/runs` for ingest

- `GET /metrics` in Prometheus text format



The dashboard shows the latest preflight result with each check's PASS, FAIL, or

STALE status and measured value, the 30-day calibration countdown with a clear

state as it approaches and passes expiry, scan run history with a drill-down into

each step's target and confirmed bed position, the device registry, and a trends

chart of reruns and the false-versus-real failure rate.



Bring up the whole stack against the simulators with Docker. This builds the API

daemon, the simulators, and the dashboard, runs the real preflight engine and

orchestrator to seed the store, and serves the dashboard on port 5173 and the API

on port 8080:



```sh

docker compose up --build

```



## Operations



The daemon is built to be run and operated, not just demoed:



- **Health and readiness.** `GET /healthz` is the liveness probe and `GET

  /readyz` pings the store for readiness, so an orchestrator routes traffic only

  once the daemon can serve data.

- **Graceful shutdown.** On SIGINT or SIGTERM the daemon drains in-flight

  requests and closes the store cleanly rather than being killed mid-write.

- **Structured logging.** The daemon logs JSON to stderr with typed fields.

- **Input validation.** Ingest bodies are size-capped and rejected when malformed;

  the `limit` query parameter is bounded so a request cannot pull an unbounded

  result set.

- **Container stack.** The compose services declare health checks and ordered

  startup; the runtime image runs as a non-root user with a `HEALTHCHECK`.



See `docs/OPERATIONS.md` for the full operations guide and `SECURITY.md` for the

trust model and how to report a vulnerability. There is no live hosted deployment:

the platform runs on a lab host or cluster behind the site's network controls.



## Requirements



- Go 1.23 or newer

- Python 3.11 or newer

- Node 20 or newer (for the dashboard)



## How to run



Build everything and run the test suites:



```sh

make build

make test

```



Run the end-to-end preflight demo against the simulated scanner. This boots the

sim scanner on a local port, points the preflight engine at it, prints a

PASS/FAIL summary, and writes the JSON and JUnit reports:



```sh

make preflight

```



Run the Go orchestrator against the simulated scanner (queries identity and one

detector channel, writes a JSON run log):



```sh

make sim &        # start the simulated scanner

make run-core     # connect, query, write runlog

```



See `docs/ARCHITECTURE.md` for how the components talk to each other and

`docs/GLOSSARY.md` for domain terms.



## License



MIT. See `LICENSE`.