{"id":37298517,"url":"https://github.com/cucapra/turnt","last_synced_at":"2026-01-17T03:01:40.976Z","repository":{"id":35095129,"uuid":"190230003","full_name":"cucapra/turnt","owner":"cucapra","description":"simple snapshot-style integration testing for commands","archived":false,"fork":false,"pushed_at":"2025-05-29T21:32:52.000Z","size":194,"stargazers_count":72,"open_issues_count":6,"forks_count":5,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-11-04T12:26:15.188Z","etag":null,"topics":["tap","testing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cucapra.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-06-04T15:33:07.000Z","updated_at":"2025-09-22T18:22:01.000Z","dependencies_parsed_at":"2023-09-24T06:16:03.831Z","dependency_job_id":null,"html_url":"https://github.com/cucapra/turnt","commit_stats":{"total_commits":148,"total_committers":4,"mean_commits":37.0,"dds":0.06756756756756754,"last_synced_commit":"7be112833c5aa9661fc3d65b089c5f05f8a6ca2f"},"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"purl":"pkg:github/cucapra/turnt","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cucapra%2Fturnt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cucapra%2Fturnt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cucapra%2Fturnt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cucapra%2Fturnt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cucapra","download_url":"https://codeload.github.com/cucapra/turnt/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cucapra%2Fturnt/sbom","scorecard":{"id":312305,"data":{"date":"2025-08-11","repo":{"name":"github.com/cucapra/turnt","commit":"b63b84d57f3d13076cac074411fdbccd0523af53"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":4.1,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Maintained","score":6,"reason":"8 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 6","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":1,"reason":"Found 2/11 approved changesets -- score normalized to 1","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/ci.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:15: update your workflow using https://app.stepsecurity.io/secureworkflow/cucapra/turnt/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:16: update your workflow using https://app.stepsecurity.io/secureworkflow/cucapra/turnt/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:27: update your workflow using https://app.stepsecurity.io/secureworkflow/cucapra/turnt/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:28: update your workflow using https://app.stepsecurity.io/secureworkflow/cucapra/turnt/ci.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:39: update your workflow using https://app.stepsecurity.io/secureworkflow/cucapra/turnt/ci.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:41: update your workflow using https://app.stepsecurity.io/secureworkflow/cucapra/turnt/ci.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:45: update your workflow using https://app.stepsecurity.io/secureworkflow/cucapra/turnt/ci.yml/main?enable=pin","Warn: pipCommand not pinned by hash: .github/workflows/ci.yml:19","Warn: pipCommand not pinned by hash: .github/workflows/ci.yml:31","Info:   0 out of   5 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   2 third-party GitHubAction dependencies pinned","Info:   0 out of   2 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'main'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 27 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-17T23:23:14.154Z","repository_id":35095129,"created_at":"2025-08-17T23:23:14.154Z","updated_at":"2025-08-17T23:23:14.154Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28492593,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T02:39:23.645Z","status":"ssl_error","status_checked_at":"2026-01-17T02:34:19.649Z","response_time":85,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["tap","testing"],"created_at":"2026-01-16T02:42:32.403Z","updated_at":"2026-01-17T03:01:40.951Z","avatar_url":"https://github.com/cucapra.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Tiny Unified Runner N' Tester (Turnt)\n=====================================\n\nTurnt is a simple snapshot testing tool inspired by [Cram][] and [LLVM's lit][lit].\nIt's good for testing things that translate text files to other text files, like compilers.\nThe idea is that each test is one input file, and you want to run a command and check that it still matches the saved output file.\n\nTo use it:\n\n1. *Configure.*\n   Decide what command you want to test.\n   Make a `turnt.toml` config file and put `command = \"mycmd {filename}\"` in it to pass each test file as an argument to `mycmd`.\n2. *Create a test.*\n   Just write an input file next to your `turnt.toml`.\n   We'll call it `foo.t`.\n3. *Take a snapshot.*\n   Run `turnt --save foo.t` to execute `mycmd foo.t` and save the standard output into `foo.out`.\n   You might want to take a look at this output to make sure it's what you expect.\n   Then you check both the input `foo.t` and output `foo.out` into version control.\n4. *Test your work.*\n   Now that you have a test in place, keep working.\n   Use `turnt *.t` to run all your tests and confirm that the output still matches.\n   If there's a mismatch, you can do `turnt --diff` to see the changes.\n   (Or if you're confident, try `turnt --save` followed by `git diff`.)\n\nTurnt's philosophy is to minimize the effort it takes to write new tests so you can quickly build up lots of them.\nYou don't write any custom logic to check results; you just record the complete \"golden\" output for each test.\n\nCompared to ordinary unit testing, \"snapshot\" tests incur the mental effort of manually inspecting diffs when things change.\nIn return, it's easier to expand test coverage.\nSnapshots also act as a crude form of documentation because every test is a complete, valid input to your program.\n\n[cram]: https://bitheap.org/cram/\n[lit]: https://llvm.org/docs/CommandGuide/lit.html\n\n\nInstall\n-------\n\nThis is a Python 3 tool.\nInstall it with [uv][]:\n\n    $ uv tool install turnt\n\nOr copy [`turnt.py`](turnt.py) directly into your project.\n\nOr, if you want to work on Turnt, you can install [Flit][], clone this repository, and type this to get a \"live\" installation with a symlink:\n\n    $ flit install --symlink --user\n\n[uv]: https://docs.astral.sh/uv/\n[flit]: https://flit.readthedocs.io/\n\n\nConfiguring\n-----------\n\nTurnt looks for a configuration file called `turnt.toml` in any of the ancestor directories of your test file.\nIt can be alongside the test file or in any containing directory.\nIt's a [TOML][] file that looks like something this:\n\n    command = \"mycmd {args} \u003c {filename}\"\n    return_code = 42\n    output.txt = \"result.txt\"\n\n### `command`\n\nSet `command` to a shell command to run on a given test file.\nThis is the only setting that is truly required.\n\nThe command is a [template][str.format]; Turnt will fill in these values:\n\n- `{filename}`: The path to the test file, relative to the working directory.\n- `{base}`: The basename of the test file, with the extension removed.\n- `{args}`: Some extra arguments that the test or user provides.\n  (See the `ARGS:` override and the `--args` command-line option below.)\n\nThe working directory for the command is the location of the `turnt.toml` configuration file, if any.\nIf there's no configuration file, then it's the location of the test file itself.\n\n### `return_code`\n\nBy default, Turnt expects the test command to succeed, i.e., exit with status code 0.\nSet `return_code` to a different status if you expect failure.\n\n### `output`\n\nBy default, Turnt captures the standard output stream from your test command.\nIf your command produces other output files \"on the side\" or you want to capture the standard error stream, you can configure the `output` table.\n\n`output` is a mapping from *snapshot extensions* to *collected filenames*.\nFor example, this TOML configuration:\n\n    output.txt = \"result.txt\"\n\nmeans that running the command will produce a file called `result.txt`, and we want to save that file in a snapshot called `\u003ctest-name\u003e.txt`.\n\nIn place of a filename, use `-` to indicate the command's standard output and `2` to indicate its standard error.\nThe default behaves like this configuration:\n\n    output.out = \"-\"\n\nwhich captures stdout and saves it in `\u003ctest-name\u003e.out`.\nDefining `output` in `turnt.toml` disables this default behavior; you can include it explicitly if you want it alongside other outputs.\n\n### `binary`\n\nBy default, Turnt looks inside test files for overrides (see below).\nThis won't work if your test inputs are binary (non-text) files (Turnt will warn you and proceed with no overrides).\nSet `binary = true` to suppress this search for overrides altogether.\n\n### `out_dir`\n\nOptionally put expected test outputs in a different directory.\nBy default, this is `.`, i.e., expected output files go \"next to\" test files.\nSet this to a different directory, relative to the test file, to keep them somewhere else.\n\n### `todo`\n\nMark a test as an allowed failure.\nYou might add a test that exposes a bug that you haven't fixed yet, for example, but you want your CI to stay \"green.\"\nSet `todo = true` to allow the test to fail; Turnt will mark it with `# todo` in the TAP output.\n\n\nPer-Test Overrides\n------------------\n\nSometimes you need to alter the setup for a specific test file.\nTurnt looks for some overrides embedded in the test file itself: for example, you might put them in a comment at the top of a test program.\n\nPut these things into your test file to override the configuration:\n\n- `CMD: \u003ccommand\u003e` overrides `command` from the configuration.\n- `ARGS: \u003carguments\u003e` adds arguments to a configured command.\n  Turnt puts this string in where the command uses `{args}`.\n- `OUT: \u003cext\u003e \u003cfilename\u003e` overrides `output` from the configuration.\n  You can specify multiple files this way: one line per file.\n- `RETURN: \u003ccode\u003e` overrides the expected exit status.\n- `TODO: \u003ctrue|false\u003e` can mark a test as a \"todo\" (allowed failure).\n\n\nMultiple Environments\n---------------------\n\nTurnt is mostly about running one command on many input files.\nSometimes, however, you need to run several commands on each file.\nThis can be especially useful for *[differential testing][dt]:*\nwhen you want to check that multiple things behave the same way by checking that they produce the same input when you give them the same input.\n\nYou can create multiple environments in your configuration file under the `envs` table.\nThe table maps environment *names* to sets of configuration options that are just like the [top-level configuration described above](#configuring).\nFor example:\n\n    [envs.baseline]\n    command = \"interp -g {filename}\"\n\n    [envs.optimized]\n    command = \"compile -O3 {filename} ; ./a.out\"\n\nEach environment can have the full complement of configuration options described above:\nfor example, `command`, `output`, and `return_code`.\nWhen you run `turnt` on some files, it will run all the environments on all the files.\nThere is also a Boolean `default` flag to turn off an environment by default; that way, you can only use it by asking for it with the `-e` flag (see below).\n\nThis example is a differential testing setup because both environments share the same (default) snapshot file:\na test `foo.t` will look for its stdout snapshot in `foo.out`.\nIf the two environments don't match, at least one will fail.\nIf you want a more standard (non-differential) setup, just set the `output` configuration differently for the two environments, like this:\n\n    [envs.interp]\n    command = \"interp -g {filename}\"\n    output.res = \"-\"\n\n    [envs.profile]\n    command = \"profile {filename}\"\n    output.prof = \"-\"\n\n[dt]: https://en.wikipedia.org/wiki/Differential_testing\n\n\nDirectory Tests\n---------------\n\nA Turnt test is usually just a single input file, but you can also organize multiple related files into a directory.\nUse the directory the same way as you would a single file:\npass its path to the `turnt` command, and the path will appear as the `{filename}` for the configured command.\nSo you might configure your test command like this:\n\n    command = \"mycmd {filename}/test.c\"\n\nif you want each test directory to contain a file called `test.c`.\n\nTurnt puts snapshots *inside* the test directory instead of adjacent to it.\nIt names them `out.\u003cextension\u003e` in that directory.\n\nThere are two configuration options just for dealing with directory tests:\n\n- `out_base`.\n  The basename for output files in directory tests: by default, `out`.\n- `opts_file`.\n  The filename to read inside of a directory test to search for embedded overrides.\n  In our above example, you could set this to `test.c` to look for `ARGS:` and friends in that file, or `opts.txt` to look for them in a separate file on the side.\n\n[toml]: https://github.com/toml-lang/toml\n[str.format]: https://docs.python.org/3/library/string.html#formatstrings\n\n\nCommand-Line Interface\n----------------------\n\nThe most common `turnt` command-line options you'll need while running and updating tests are:\n\n- `--save`: Bless the current output from each test as the \"correct\" output, saving it to the output file that you'll want to check into version control.\n- `--diff`: Show diffs between the actual and expected output for each test.\n  (The diff goes to stderr while the [TAP][] results remain on stdout.)\n\nYou also might enjoy:\n\n- `--parallel` or `-j`: Run your tests faster using parallel threads.\n\nThese options are useful when working with one specific test file:\n\n- `--verbose` or `-v`: Disable Turnt's default behavior where it will suppress test commands' stderr output. The result is more helpful but harder to read.\n- `--print` or `-p`: Instead of checking test results, just run the command and show the output directly. This can be useful (especially in combination with `-v`) when iterating on a test interactively.\n- `--args` or `-a`: Override the `{args}` string in the test command.\n\nThese options lets you switch between different test environments:\n\n- `--env` or `-e`: Give the name of a configured [environment](#multiple-environments) to run. Use this multiple times to run multiple environments. By default, Turnt runs all the configured environments for every test. Use the special name `default` for the top-level, unnamed test environment.\n- `--config` or `-c`: Look for this config filename instead of the default `turnt.toml`.\n\n\nTAP\n---\n\nTurnt outputs results in the machine-readable [TAP][] format.\nTo make the output more pleasant to read, you can pipe it into a tool like [tap-difflet][], [tap-dot][], or [faucet][]:\n\n    $ npm install -g tap-difflet\n    $ turnt *.t | tap-difflet\n\n[tap]: http://testanything.org\n[tap-difflet]: https://github.com/namuol/tap-difflet\n[tap-dot]: https://github.com/scottcorgan/tap-dot\n[faucet]: https://github.com/substack/faucet\n\n\nCredits\n-------\n\nTurnt is by [Adrian Sampson][adrian] and [Alexa VanHattum][alexa].\nWe made it to test various research compilers in [Capra][].\nThe license is [MIT][].\n\n[adrian]: https://www.cs.cornell.edu/~asampson/\n[alexa]: https://www.cs.cornell.edu/~avh/\n[capra]: https://capra.cs.cornell.edu\n[mit]: https://opensource.org/licenses/MIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcucapra%2Fturnt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcucapra%2Fturnt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcucapra%2Fturnt/lists"}