https://github.com/degory/ghul-test
ghūl compiler integration test runner
https://github.com/degory/ghul-test
compiler ghul test test-automation testing
Last synced: 26 days ago
JSON representation
ghūl compiler integration test runner
- Host: GitHub
- URL: https://github.com/degory/ghul-test
- Owner: degory
- License: mit
- Created: 2020-12-28T18:14:07.000Z (over 5 years ago)
- Default Branch: main
- Last Pushed: 2026-05-30T06:46:26.000Z (about 1 month ago)
- Last Synced: 2026-05-30T08:16:45.337Z (about 1 month ago)
- Topics: compiler, ghul, test, test-automation, testing
- Language: Shell
- Homepage: https://ghul.dev
- Size: 290 KB
- Stars: 1
- Watchers: 0
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
# ghūl compiler integration test runner
[](https://github.com/degory/ghul-test/actions?query=workflow%3ACICD+branch%3Amain)
[](https://www.nuget.org/packages/ghul.test/)
[](https://github.com/degory/ghul-test/releases)
[](https://github.com/degory/ghul-test/releases)
[](https://github.com/degory/ghul-test/issues)
[](https://github.com/degory/ghul-test/blob/main/LICENSE)
[](https://ghul.dev)
This is a very simple snapshot based test runner which is used by the [ghūl programming language](https://ghul.dev) [compiler](https://github.com/degory/ghul) [integration tests](https://github.com/degory/ghul/tree/master/integration-tests). It compares test expectations, in the form of snapshot text files, against the actual outputs of the compiler and test executables and flags any differences.
## Test Folder Structure
A test directory must contain at least two things:
- **One or more `.ghul` source files** – the sources to compile.
- A `ghulflags` file – flags passed directly to the compiler when building the test.
Any directory containing a `ghulflags` file is treated as a test. Subdirectories without this file are ignored by the queue logic.
Optional expectation and configuration files may also be present:
| File | Purpose |
| --- | --- |
| `fail.expected` | If present, the build is expected to fail. Its mere presence enables this behaviour; the file contents are ignored. |
| `err.expected` | Expected compiler error output. Actual errors are extracted from `compiler.out`, sorted, and diffed against this file. |
| `warn.expected` | Expected compiler warning output. Warnings undergo the same grep and sort process as errors. |
| `run.expected` | Expected stdout from running the compiled binary. |
| `il.expected` | Expected IL disassembly output (from the `il.out` file). |
| `ghulflags` | Mandatory file containing additional command line flags for the compiler. |
| `disabled*` | Any file beginning with `disabled` causes the test to be skipped. |
A basic “hello world” example can be found in the `integration-tests` folder of this repository.
## Expectation Comparison Workflow
1. The runner invokes the compiler using the arguments from `ghulflags` and the test’s `.ghul` sources. Compiler stdout/stderr is written to `compiler.out`.
2. `grep` extracts error and warning lines from `compiler.out` into `err.grep` and `warn.grep` respectively.
3. These files are sorted with `sort` (with `LC_COLLATE` set to `C` for stable output) into `err.sort` and `warn.sort`.
4. `diff` compares `err.sort` to `err.expected` and `warn.sort` to `warn.expected`. Whitespace differences are ignored and carriage returns are stripped.
5. If compilation succeeded, `ghul-runtime.dll` is symlinked into the test directory and the binary is executed via `dotnet`. Output is captured in `run.out` and compared to `run.expected`.
6. If an `il.expected` file exists, `diff` is run against the generated `il.out` file as well.
Any mismatches cause a failure report containing a unified diff of the actual versus expected output.
## Command Line Usage
```text
ghul-test [--use-dotnet-build] [--runtime-dll ] [...]
```
- `--use-dotnet-build` – expects each test folder to be an MSBuild project. For ghūl projects the file should end with `.ghulproj`. The runner builds the project with `dotnet build` instead of invoking the compiler directly.
- `--runtime-dll ` – use the supplied `ghul-runtime.dll` for compiled test binaries instead of the version that ships with `ghul-test`. The path must point to an existing file. Takes precedence over the `GHUL_RUNTIME_DLL` environment variable. Has no effect under `--use-dotnet-build`, which resolves the runtime via the test project's own `PackageReference`.
- `` – one or more directories containing tests. Each is recursively searched for subdirectories with a `ghulflags` file if not using `--use-dotnet-build`.
Environment variables influence behaviour:
- `HOST` and `TARGET` – specify the CLI used to run the compiler and the compiled binary (default `dotnet`).
- `CI` – when set to `1` or `true`, enables CI mode. In this mode `ghul-runtime.dll` is taken from the test runner's own location unless overridden by `--runtime-dll` / `GHUL_RUNTIME_DLL`.
- `GHUL_RUNTIME_DLL` – path to a `ghul-runtime.dll` to use for compiled test binaries, overriding the version that ships with `ghul-test`. Equivalent to passing `--runtime-dll`; the CLI flag wins if both are set.
- `TEST_PROCESSES` – number of worker processes to use. If unset, a value derived from CPU count is used.
The runner prints progress for each test and a final summary indicating total, enabled, passed and failed counts.
## Runtime Library Handling
When the compiler is invoked directly (the default and CI modes), the produced executable expects to find `ghul-runtime.dll` beside it. The runner therefore creates a symbolic link in the test directory pointing to the runtime library. This link is not needed when `--use-dotnet-build` is used. After the test completes successfully, the link is deleted during cleanup.
By default the runtime DLL is sourced from the published compiler's directory (LOCAL mode) or the test runner's own install directory (CI mode). When the integration tests need to run against a runtime version other than the one `ghul-test` itself was packaged with — for example, when CI builds a compiler that depends on a newer `ghul.runtime` than the pinned `ghul.test` ships with — pass `--runtime-dll ` or set `GHUL_RUNTIME_DLL` to override the discovered location with an explicit DLL path.
## MSBuild Projects
This runner does **not** execute arbitrary MSBuild projects. It either drives the compiler directly on `.ghul` source files or, when `--use-dotnet-build` is supplied, assumes the folder already contains a valid MSBuild project (for ghūl projects this means a `*.ghulproj` file). Only a small set of standard .NET assemblies is referenced so complex projects are out of scope.
## Dependencies
The runner relies on several standard Unix utilities being available in the environment: `grep`, `sort`, `diff` and `ln`. A .NET 8 SDK installation is required; mono is not supported.
## Writing New Tests
This repository includes helper scripts under `./scripts`:
- `create.sh` – create a new test directory from the built-in template.
- `capture.sh` – update expectation files after running a test.
1. Run `./scripts/create.sh` and provide the new test name.
2. Edit the generated `.ghul` sources and `ghulflags` as required.
3. Execute `ghul-test ` (expect it to fail initially). The runner produces `.out` files with the actual output.
4. Run `./scripts/capture.sh ` to copy the `.out` files over the corresponding `*.expected` files.
5. Re-run `ghul-test` and verify the test now passes.
6. Commit the test directory along with the expectations.
Refer to the [ghūl compiler integration tests](https://github.com/degory/ghul/tree/main/integration-tests) for many real‑world examples of this structure.