https://github.com/anko/txm

Markdown code example tester; language-agnostic
https://github.com/anko/txm

documentation language-agnostic markdown shell tap testing

Last synced: 7 days ago
JSON representation

Markdown code example tester; language-agnostic

• Host: GitHub
• URL: https://github.com/anko/txm
• Owner: anko
• License: isc
• Created: 2015-04-27T19:59:20.000Z (over 9 years ago)
• Default Branch: master
• Last Pushed: 2023-07-18T21:17:32.000Z (over 1 year ago)
• Last Synced: 2024-10-22T10:02:34.600Z (17 days ago)
• Topics: documentation, language-agnostic, markdown, shell, tap, testing
• Language: JavaScript
• Homepage:
• Size: 492 KB
• Stars: 44
• Watchers: 3
• Forks: 0
• Open Issues: 5
• Metadata Files:
  • Readme: readme.markdown
  • License: LICENSE

Awesome Lists containing this project

README

        
# txm [![](https://img.shields.io/npm/v/txm.svg)][1] [![](https://img.shields.io/github/actions/workflow/status/anko/txm/ci.yml?branch=master)][2] [![](https://img.shields.io/coveralls/github/anko/txm)][coveralls]



Command-line tool that checks correctness of your [Markdown][markdown]

documentation's code examples.  Parses `` annotations

preceding code blocks, runs them, and checks that the outputs match.

 - __Only uses HTML comments__

   
The annotations aren't rendered.  You retain formatting

   control.

 - __Works with any programming language__

   
You choose the shell command(s).  Many languages in the same doc

   are OK.

 - __Helpful failure diagnostics__

   
Colours (optional), diffs, line numbers, exit code, stderr,

   invisible characters, etc.

 - __Parallel tests on multi-core machines__

   
Configurable.  Result output ordering remains constant.

 - __[TAP][tap-spec] format output__

   
The standard supported by many testing tools.

# Example

1. Write a `README.md`, with comment annotations:

   ```markdown

   # console.log

   The [console.log][1] function in [Node.js][2] stringifies the given arguments

   and writes them to `stdout`, followed by a newline.  For example:

   

   

       console.log('a')

       console.log(42)

       console.log([1, 2, 3])

   The output is:

   

       a

       42

       [ 1, 2, 3 ]

   [1]: https://nodejs.org/api/console.html#console_console_log_data_args

   [2]: https://nodejs.org/

   ```

   See [§ *Use*](#use) for more detail on how annotations work. Fenced code

   blocks delimited by `` ``` `` work too.  Language tags also.

2. Run:

   ```bash

   $ txm README.md

   ```

3. See output:

   

   

   > ```tap

   > TAP version 13

   > 1..1

   > ok 1 simple example

   >

   > # 1/1 passed

   > # OK

   > ```

- - -

Examples of other use-cases:

Testing Node.js code with ESM imports

Running code from `node`'s stdin as an ES module requires

`--input-type=module`.

```markdown

Demonstrating that the root directory is a directory:

    import { stat } from 'fs/promises'

    console.log((await stat('/')).isDirectory())

    true

```

> ```

> TAP version 13

> 1..1

> ok 1 example

>

> # 1/1 passed

> # OK

> ```

Testing C code with gcc

Any sequence of shell commands is a valid `!test program`, so you can e.g. cat

the test input into a file, then compile and run it:

```markdown

Here is a simple example C program that computes the answer to life, the

universe, and everything:

    #include 

    int main () {

        printf("%d\n", 6 * 7);

    }

    42

```

> ```

> TAP version 13

> 1..1

> ok 1 printf

>

> # 1/1 passed

> # OK

> ```

In practice you might want to invoke `mktemp` in the `!test program` to avoid

multiple parallel tests overwrting each other's files.  Or pass `--jobs 1` to

run tests serially.

Redirecting stderr→stdout, to test both in the same

block

Prepending `2>&1` to a shell command [redirects][shell-redirection-q] `stderr`

to `stdout`.  This can be handy if you don't want to write separate `!test out`

and `!test err` blocks.

```markdown

    console.error("This goes to stderr!")

    console.log("This goes to stdout!")

    This goes to stderr!

    This goes to stdout!

```

> ```

> TAP version 13

> 1..1

> ok 1 print to both stdout and stderr

>

> # 1/1 passed

> # OK

> ```

Ignoring the tested program's exit code

Normally, you'd use `!test exit nonzero` (or a specific exit code) to tell txm

that a test is expected to fail.  But since you need to write that before every

failing run, it can get pointlessly repetitive if e.g. it's obvious only from

the output of your program when it failed.

In such cases, just put `|| true` after the program command to make the shell

swallow the exit code and pretend to `txm` that it was `0`.  Remember that the

program tests are run with can be a whole script.

```markdown

    console.log("Hi before throw!")

    throw new Error("AAAAAA!")

    Hi before throw!

```

> ```

> TAP version 13

> 1..1

> ok 1 don't fail

>

> # 1/1 passed

> # OK

> ```

Testing examples that call assert

If your example code calls `assert` or such (which throw an error and exit

nonzero when the assert fails), then you don't really need an output block,

because the example already documents its assumptions.

In such cases you can use use a `!test check` annotation.  This simply runs the

code, ignoring its output.

```markdown

    const assert = require('assert')

    assert(1 + 1 == 2)

```

> ```

> TAP version 13

> 1..1

> ok 1 laws of mathematics

>

> # 1/1 passed

> # OK

> ```

If you are using an assert library that can output ANSI colour codes, it should

detect that it is running without a TTY (as tests do), and not output colour.

But if txm itself is run in coloured mode, the `TXM_HAS_COLOUR` environment

variable will be set to `1`, and it's safe to force colour output on; they will

be included in txm's error output.

As you may be suspecting, this readme is itself tested with txm.  All of the

above examples run as part of the automatic tests, locally and [on the CI

server](https://github.com/anko/txm/actions/workflows/ci.yml?query=is%3Asuccess).

If you want to see the comment annotations, [see the readme

source](https://github.com/anko/txm/blob/master/readme.markdown?plain=1).

(It's a little trippy, because txm is recursively running itself.)

# Install

To install for current directory's project: `npm install txm`


To install globally: `npm install -g txm`

Requires [Node.js][nodejs] (minimum version tested is _current LTS_).

# Use

## Command line

### `txm [--jobs ] [filename]`

 - `filename`: Input file (default: read from `stdin`)

 - `--jobs`: How many tests may run in parallel. (default: `os.cpus().length`)

   When a test finishes, txm will only print its output after all

   earlier-defined tests have printed their outputs, so that results appear in

   the same order tests were defined.  Further tests continue to run in the

   background, regardless of how many results are pending print.

 - `--version`

 - `--help`

## Annotations

HTML comments that start with `!test` are read specially.  Use a separate

comment for each annotation.

 - #### `!test program `

   The `` is run as a shell command for each following matching

   input/output pair.  It gets the input on `stdin`, and is expected to produce

   the output on `stdout`.  The program may be as many lines as you like; a

   full shell script if you wish.

   The declared program is used for all tests after here, until a new program

   is declared.

 - #### `!test in ` / `!test out ` / `!test err `

   The next code block is read as the input to give to a program for the test

   ``, or expected stdout or stderr of the test ``.  These are

   matched by ``, and may be anywhere in relation to each other.

   Errors are raised if a test has no input (`in`) or no output (`out` nor

   `err`), or if it has duplicates of any.

 - #### `!test check `

   The next code block is read as a check test.  The program gets this as

   input, but its output is ignored.  The test will pass if the program exits

   successfully.  (With exit code `0`, or that specified in a `!test exit`

   command prior.)

   Use this for code examples that check their own correctness, for example by

   calling an `assert` function.

 - #### `!test exit `


   The _next test_ which is fully read is expected to fail, and to exit with

   the given `code`, instead of the default `0`.

   You can use `!test exit nonzero` to accept any non-0 exit code.

 - #### `!test only`

   If any test has this command in front of it, all tests without it are

   skipped.  (They don't run, and their output is suppressed.)

   This is intended for developer convenience:  When you have lots of tests of

   which only a few are failing, you can use this command to focus on them, so

   other tests don't waste time running or clutter your screen.

## Behaviour details

### Exit code

`txm` exits `0` if and only if all tests pass.

### Invisible characters

In diff additions and deletions, [C0 Control Characters][control-chars] (such

as Null, Line Feed, or Space), which are ordinarily invisible, are shown as the

corresponding [Unicode Control Picture][control-picture]. These take the form

of small diagonally arranged letters, so Null becomes ␀, Line Feed becomes ␊,

and Space becomes ␠. This is the standard way to show this set of invisible

characters.

Whenever such characters are used, an index will be present in the accompanying

test data, listing what original character each picture corresponds to, with

its name, C escape, and Unicode code point. This is intended to give as much

information as possible, because bugs relating to invisible characters are

awkward to debug.

If an invisible character is not part of the diff, it is shown normally

(without a Control Picture replacement.)

To maintain line breaks, the Line Feed character is kept as-is, with its

Control Picture (␊) added at the end of the line for clarity.

Invisible characters that aren't part of the C0 set are shown as-is. Examples

include the zero-width space, or right-to-left text flow marker.

[control-chars]: https://en.wikipedia.org/wiki/C0_and_C1_control_codes#C0_controls

[control-picture]: https://en.wikipedia.org/wiki/Unicode_control_characters#Control_pictures

### Colour (color, for Americans grepping)

Coloured output is automatically enabled when outputting directly to a

colour-capable console interface, and disabled otherwise.  It can be forced on

or off with the environment variables `NO_COLOR=1` or `FORCE_COLOR=1`, or with

the options `--no-color` or `--color`.

Stripping colour codes from coloured output does not change its logical

meaning, and indeed the same text is emitted regardless of whether colour is

enabled.  The colours do not themselves carry meaning; they're just hints to

guide the eye.

### HTML comment character restrictions

The [HTML spec regarding comments][html-comments-spec] has a few restrictions

on what comments may contain:

> the text must not start with the string `>`, nor start with the string `->`,

> nor contain the strings ``, or `--!>`, nor end with the string

> ` -->` is legal HTML, and will be parsed by txm as the command `!test in

-->`.

To write literally `#-`, write `##-` instead, and so on.  `#` acts normally

everywhere else, and doesn't need to be escaped.

### Environment variables

For advanced use, your test program sees the same environment variables that

txm sees, plus these introduced by txm:

- `TXM_INDEX` (1-based number of test)

- `TXM_NAME` (name of test)

- `TXM_INDEX_FIRST`, `TXM_INDEX_LAST` (indexes of first and last tests that

  will be run)

- `TXM_INPUT_LANG` (the [language identifier][gh-markdown-lang] of the

  input/check markdown code block, if any)

- `TXM_HAS_COLOUR`, `TXM_HAS_COLOR` (both set to `1` if outputting with colours

  enabled, or to `0` if disabled; they are logically equivalent, just alternate

  spellings)

You can use these for example to descriptively name log files, or to easily

detect languages and test them differently.

You can also tell with the colour variables whether txm is doing coloured

output or not, and have your program emit debug output with ANSI colour codes

by your method of choice.  This is probably only reasonable to do for the

output of `check` tests, which output is shown unmodified.  Don't do this for

`in`/`out` tests unless you _really_ know what you're doing; the colour codes

used by txm's automatic diffing will interfere, and you'll get garbage.

# Trivia

The name txm stands for "tests ex markdown" as in "deus ex machina", or

*temptamentum ex Markdown* I guess if you're feeling extra Latin.

# License

[ISC](LICENSE)

[1]: https://www.npmjs.com/package/txm

[2]: https://github.com/anko/txm/actions/workflows/ci.yml?query=branch%3Amaster

[coveralls]: https://coveralls.io/github/anko/txm

[gh-markdown-lang]: https://docs.github.com/en/github/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks#syntax-highlighting

[html-comments-spec]: https://html.spec.whatwg.org/multipage/syntax.html#comments

[markdown]: http://daringfireball.net/projects/markdown/syntax

[nodejs]: https://nodejs.org/

[shell-redirection-q]: https://superuser.com/questions/1179844/what-does-dev-null-21-true-mean-in-linux

[tap-spec]: https://testanything.org/tap-version-13-specification.html