Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/manticoresoftware/clt

Command Line Tester
https://github.com/manticoresoftware/clt

Last synced: 9 months ago
JSON representation

Command Line Tester

Awesome Lists containing this project

README 

          
# Command Line Tester (CLT)



A powerful testing tool that allows you to execute and record commands inside a Docker container, replay them, and validate whether your results match expected outputs. CLT helps automate testing of command-line applications with consistent environments.



## Features



- Record interactive command sessions in reproducible test files

- Replay recorded sessions to verify consistent behavior

- Intelligent output comparison with regex pattern matching

- Reusable test blocks for modular testing

- GitHub Action support for CI/CD integration

- Cross-platform compatibility via Docker containers



## Installation



Installation is straightforward. Simply clone the repository and start using it:



```bash

git clone https://github.com/manticoresoftware/clt.git

cd clt

./clt help

```



### Prebuilt Binaries



CLT provides prebuilt binaries for Linux environments (both amd64 and arm64), which are automatically detected based on your platform. While binaries for macOS and Windows are not currently available, they can be built if required.



NOTE: CLT requires Docker to be installed on your machine as all tests run in a container environment.



## Usage



### Recording Tests



1. Start recording a test session with the `record` command:



   ```bash

   ./clt record centos:7

   ```



   This starts a recording session using the specified Docker image. You'll see the path to the recording file displayed:



   ```bash

   Recording data to file: ./tests/centos:7_20230714_161043.rec

   ```



2. Perform your commands in the interactive shell. All inputs and outputs will be recorded.



3. When finished, press `^D` (Ctrl+D) once to stop recording and save your test.



### Replaying and Validating Tests



To validate and replay a recorded test:



```bash

./clt test -t ./tests/centos:7_20230714_161043.rec -d centos:7

echo $?

```



- `-t` specifies the test file to replay

- `-d` enables debug output that shows diff information if results don't match



The exit code will be 0 if the test passed (outputs match) or 1 if there are differences.



## Web-based UI



CLT includes a web-based user interface for editing and running tests. The UI provides an intuitive interface for managing test files, with real-time pattern matching and diff comparison. To learn more about the UI, its features, and how to set it up, see the [UI documentation](ui/README.md).



Key UI features include:

- File tree explorer for navigating test files

- Real-time comparison with pattern matching

- Docker image configuration for test validation

- GitHub authentication for access control



We utilize bash to initiate an interactive environment when you record a test. It's important to note that we reset the environment to ensure maximum compatibility with various operating systems. As of now, there is no option to pass environment variables from outside into the test environment.



### Refining Tests



After recording, you can refine your test to handle dynamic outputs:



```bash

./clt refine -t ./tests/centos:7_20230714_161043.rec centos:7

```



This will open your configured editor to modify test outputs and add patterns where needed.



## Test File Syntax



CLT test files (`.rec`) use a special syntax to differentiate between commands, outputs, and other sections.



### Supported Sections



**Note:** CLT uses en dashes (`–`) in its section markers, not regular hyphens (`-`). The code samples below show the correct syntax with en dashes.



| Section | Syntax | Description |

|---------|--------|-------------|

| `Input` | `––– input –––` | Marks the beginning of a command input section |

| `Output` | `––– output –––` | Marks the beginning of a command output section |

| `Output with Checker` | `––– output: checker-name –––` | Like output but uses a custom checker program |

| `Block` | `––– block: path/to/block –––` | Includes a reusable test block from a `.recb` file |

| `Comment` | `––– comment –––` | Marks a comment section (ignored during test execution) |



**Note:** The `duration` section (`––– duration: 123ms (45.67%) –––`) is automatically generated by the test system and should not be added manually to `.rec` files. It appears in the resulting `.rep` files after test execution.



### Basic Structure



A test file typically consists of repeating input and output sections:



```

––– input –––

echo "Hello, world!"

––– output –––

Hello, world!

```



Each input section contains commands exactly as they were typed, and each output section contains the expected output that should be matched during test replay.



## Pattern Matching



For outputs that change between runs (timestamps, PIDs, etc.), CLT supports regex pattern matching.



### Using Regular Expressions



Place regex patterns between `#!/` and `/!#` markers in your test output:



```

#!/[0-9]+/!#  # Matches any number

```



### Using Named Patterns



CLT provides predefined patterns in the `.clt/patterns` file. Use them with the syntax `%{PATTERN_NAME}`:



```

%{SEMVER}  # Matches semantic version numbers

```



You can define custom patterns by adding them to your project's `.clt/patterns` file using the format:



```

PATTERN_NAME REGEX_PATTERN

```



Example `.clt/patterns` file:



```

SEMVER [0-9]+\.[0-9]+\.[0-9]+

YEAR [0-9]{4}

IPADDR [0-9]+\.[0-9]+\.[0-9]+\.[0-9]+

COMMITDATE [a-z0-9]{7}@[0-9]{6}

```



**Note:** CLT first loads the global patterns from `$PROJECT_DIR/.clt/patterns` and then appends your project-specific patterns from `.clt/patterns` for extended functionality.



## Reusable Blocks



Create modular test components by extracting flows into `.recb` files placed in the same directory as your test files:



```

––– block: login-sequence –––

```



This will include the contents of the `login-sequence.recb` file located in the same directory as your test. You can also organize blocks in subdirectories using relative paths:



```

––– block: auth/admin-login –––

```



The path is always relative to the location of the `.rec` file that's including the block. Blocks can be nested, allowing you to compose complex test scenarios from reusable components.



### Block Internal Steps (UI Feature)



When using the CLT UI, you can add steps directly inside block references. These internal steps are automatically saved to the corresponding `.recb` file:



- **Block with internal steps**: Steps added inside a block in the UI are saved to the `.recb` file

- **Path resolution**: Block paths like `../base/block.recb` are resolved relative to the test file directory

- **Automatic file creation**: `.recb` files are created/updated automatically when blocks contain internal steps

- **Backward compatibility**: Existing block references without internal steps continue to work as before

- **Status propagation**: Block internal steps now properly inherit status from test execution results

- **UI feedback**: Interface provides visual feedback during test runs with opacity effects



## Custom Output Checkers



For advanced output validation beyond regex patterns, CLT supports custom checker programs. These are specified in the output statement:



```

––– output: custom-checker –––

```



### How Custom Checkers Work



1. Place your checker executable in the `.clt/checkers/` directory

2. The checker will receive two arguments:

   - First argument: Path to a temp file containing expected output (from .rec file)

   - Second argument: Path to a temp file containing actual output (from replay)

3. The checker should return:

   - Exit code 0 if outputs match (test passes)

   - Non-zero exit code if there are differences (test fails)

4. Any stdout/stderr output from the checker will be displayed when there are differences



### Creating Custom Checkers



Checkers can be written in any language, but must be executable. Here's an example of a simple bash checker that verifies if a specific string exists in the output:



```bash

#!/bin/bash

# simple-contains-checker.sh - A checker that verifies if output contains a specific string



EXPECTED_FILE="$1"  # Path to expected output from .rec file

ACTUAL_FILE="$2"    # Path to actual output from test run



# Get the string to search for from the first line of expected file

SEARCH_STRING=$(head -n 1 "$EXPECTED_FILE")



# Check if the actual output contains this string

if grep -q "$SEARCH_STRING" "$ACTUAL_FILE"; then

  echo "✓ Found expected string: $SEARCH_STRING"

  exit 0  # Success

else

  echo "✗ Missing expected string: $SEARCH_STRING"

  exit 1  # Failure

fi

```



This provides great flexibility for validating complex outputs without needing regex patterns.



## Environment Variables



CLT's behavior can be customized through several environment variables:



| Variable | Description | Default |

|----------|-------------|---------|

| `CLT_DEBUG` | Enable debug mode to show detailed output | `0` |

| `CLT_DIFF_INLINE` | Display differences in-line instead of side-by-side | `0` |

| `CLT_EDITOR` | Custom editor for refining tests | Auto-detected (`nano` or `vim`) |

| `CLT_PROMPTS` | Array of additional prompts to detect (e.g., `CLT_PROMPTS=("mysql> ")`) | `("clt> ")` |

| `CLT_NO_COLOR` | Disable colored output | Not set |

| `DEFAULT_DELAY` | Default delay in ms between each command in the test | `5` |

| `CLT_RUN_ARGS` | Additional arguments to pass to `docker run` | Not set |



## GitHub Actions Integration



CLT provides a ready-to-use GitHub action to run tests in your CI/CD pipeline. You can customize the action with several parameters:



| Parameter | Description | Default |

|-----------|-------------|---------|

| `image` | Docker image to run the tests in | Required |

| `test_prefix` | Filter tests by prefix in tests folder | `tests/` |

| `run_args` | Additional arguments to pass to `docker run` | `''` |

| `init_code` | Code to execute before running tests | `''` |

| `timeout` | Allowed timeout in minutes for tests | `5` |

| `artifact` | Artifact to download for local built docker image | Not set |

| `repository` | Repository to checkout | Not set |

| `ref` | Branch name or ref to checkout | Default branch |



### Example Workflow



```yaml

name: CLT tests



on:

  push:

    branches: [ main ]

  pull_request:

    branches: [ main ]



jobs:

  clt:

    name: Run CLT tests

    strategy:

      fail-fast: false

      matrix:

        image: [ "ubuntu:bionic", "ubuntu:focal", "ubuntu:jammy", "debian:buster", "debian:bullseye", "debian:bookworm" ]

    runs-on: ubuntu-22.04

    steps:

      - uses: manticoresoftware/clt@v3

        with:

          image: ${{ matrix.image }}

          test_prefix: test/clt-tests/

          run_args: -e TELEMETRY=0

          timeout: 10

          init_code: |

            apt-get update

            apt-get install -y curl

```



## Project Structure



CLT uses a specific directory structure for its key files:



```

.clt/

├── checkers/     # Custom output checker executables

└── patterns      # Pattern definitions for regex matching

tests/

├── *.rec         # Test recording files

└── *.recb        # Reusable test blocks

```



The `.clt/` directory contains CLT configuration and special-purpose files, while the actual test files and reusable blocks are stored in the tests directory or other directories specified for your tests.



## File Extensions



CLT uses several file extensions:



| Extension | Description |

|-----------|--------------|

| `.rec` | Original record of inputs and outputs, may include links to block files |

| `.recb` | Reusable block file that can be included in `.rec` files |

| `.rep` | Replay file containing results of replaying a `.rec` file |



## Current Limitations



- Use `^D` (Ctrl+D) only once to close the CLT environment; for other exits, use the `exit` command

- Avoid using `^C`, `^V`, `^Z`, and other control characters as they may not function correctly

- Reverse search (`^R`) is currently unsupported

- Complete tests in the `clt> ` shell and press `^D` to terminate the session

- Use a simple terminal like iTerm; VS Code terminal may cause unusual behavior

- Each command in the `.rec` test must have an output that contains a newline



### Supported Keystrokes



Only the following keystrokes are fully supported:



- Standard input characters

- Left and right arrow keys

- Backspace and delete keys

- CTRL+a (beginning of line) and CTRL+e (end of line)



## Developers Section



### Building Tools



To build the `rec` and `cmp` tools for both aarch64 and amd64 Linux:



```bash

./bin/cross-build

git add .

git commit -m 'Update binaries'

```



### The Replay Flow



During test execution, CLT:



1. Finds `.rec` files and compiles them into a ready-to-use version on the fly

2. Executes each command in sequence, generating a `.rep` file

3. Uses the `cmp` tool to compare the results with the compiled version of the `.rec` files



### Using Custom Prompts



If you need to work with additional command prompts (e.g., `mysql>` prompt), configure them as follows:



```bash

CLT_PROMPTS=("mysql> ")

```



This helps CLT correctly detect command completion in different interactive shells.