Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/myint/cram

Testing framework for command-line applications
https://github.com/myint/cram

Last synced: 4 days ago
JSON representation

Testing framework for command-line applications

Awesome Lists containing this project

README 

        
======================

 Cram: It's test time

======================



.. image:: https://travis-ci.org/myint/cram.svg?branch=master

    :target: https://travis-ci.org/myint/cram

    :alt: Build status



Cram is a functional testing framework for command line applications

based on Mercurial_'s `unified test format`_.



Cram tests look like snippets of interactive shell sessions. Cram runs

each command and compares the command output in the test with the

command's actual output.



Here's a snippet from `Cram's own test suite`_::



    The $PYTHON environment variable should be set when running this test

    from Python.



      $ [ -n "$PYTHON" ] || PYTHON="`which python`"

      $ if [ -n "$COVERAGE" ]; then

      >   coverage erase

      >   alias cram="`which coverage` run --branch -a $TESTDIR/../cram.py"

      > else

      >   alias cram="$PYTHON $TESTDIR/../cram.py"

      > fi

      $ command -v md5 > /dev/null || alias md5=md5sum



    Usage:



      $ cram -h

      [Uu]sage: cram \[OPTIONS\] TESTS\.\.\. (re)



      [Oo]ptions: (re)

        -h, --help          show this help message and exit

        -V, --version       show version information and exit

        -q, --quiet         don't print diffs

        -v, --verbose       show filenames and test status

        -i, --interactive   interactively merge changed test output

        -y, --yes           answer yes to all questions

        -n, --no            answer no to all questions

        -E, --preserve-env  don't reset common environment variables

        --keep-tmpdir       keep temporary directories

        --shell=PATH        shell to use for running tests

        --indent=NUM        number of spaces to use for indentation



The format in a nutshell:



* Cram tests use the ``.t`` file extension.



* Lines beginning with two spaces, a dollar sign, and a space are run

  in the shell.



* Lines beginning with two spaces, a greater than sign, and a space

  allow multi-line commands.



* All other lines beginning with two spaces are considered command

  output.



* Output lines ending with a space and the keyword ``(re)`` are

  matched as `Perl-compatible regular expressions`_.



* Lines ending with a space and the keyword ``(glob)`` are matched

  with a glob-like syntax. The only special characters supported are

  ``*`` and ``?``. Both characters can be escaped using ``\``, and the

  backslash can be escaped itself.



* Output lines ending with either of the above keywords are always

  first matched literally with actual command output.



* Lines ending with a space and the keyword ``(no-eol)`` will match

  actual output that doesn't end in a newline.



* Actual output lines containing unprintable characters are escaped

  and suffixed with a space and the keyword ``(esc)``. Lines matching

  unprintable output must also contain the keyword.



* Anything else is a comment.



.. _Mercurial: http://mercurial.selenic.com/

.. _unified test format: http://www.selenic.com/blog/?p=663

.. _Cram's own test suite: http://bitbucket.org/brodie/cram/src/tip/tests/cram.t

.. _Perl-compatible regular expressions: http://en.wikipedia.org/wiki/Perl_Compatible_Regular_Expressions



Download

--------



* cram-0.5.tar.gz_ (22 KB, requires Python 2.4-2.7 or Python 3.1-3.2)



.. _cram-0.5.tar.gz: http://bitheap.org/cram/cram-0.5.tar.gz



Installation

------------



You can use pip_ to install Cram::



    $ pip install cram



Or you can install Cram the old fashioned way::



    $ wget http://bitheap.org/cram/cram-0.5.tar.gz

    $ tar zxvf cram-0.5.tar.gz

    $ cd cram-0.5.tar.gz

    $ python setup.py install



.. _pip: http://pypi.python.org/pypi/pip



Usage

-----



Cram will print a dot for each passing test. If a test fails, a

`unified context diff`_ is printed showing the test's expected output

and the actual output. Skipped tests (empty tests and tests that exit

with return code ``80``) are marked with ``s`` instead of a dot.



For example, if we run Cram on `its own example tests`_::



    .s.!

    --- /home/brodie/src/cram/examples/fail.t

    +++ /home/brodie/src/cram/examples/fail.t.err

    @@ -3,21 +3,22 @@

       $ echo 1

       1

       $ echo 1

    -  2

    +  1

       $ echo 1

       1



     Invalid regex:



       $ echo 1

    -  +++ (re)

    +  1



     Offset regular expression:



       $ printf 'foo\nbar\nbaz\n\n1\nA\n@\n'

       foo

    +  bar

       baz



       \d (re)

       [A-Z] (re)

    -  #

    +  @

    s.

    # Ran 6 tests, 2 skipped, 1 failed.



Cram will also write the test with its actual output to

``examples/fail.t.err``.



When you're first writing a test, you might just write the commands

and run the test to see what happens. If you run Cram with ``-i`` or

``--interactive``, you'll be prompted to merge the actual output back

into the test. This makes it easy to quickly prototype new tests.



You can specify a default set of options by creating a ``.cramrc``

file. For example::



    [cram]

    verbose = True

    indent = 4



Is the same as invoking Cram with ``--verbose`` and ``--indent=4``.



To change what configuration file Cram loads, you can set the

``CRAMRC`` environment variable. You can also specify command line

options in the ``CRAM`` environment variable.



Note that the following environment variables are reset before tests

are run:



* ``TMPDIR``, ``TEMP``, and ``TMP`` are set to the test runner's

  ``tmp`` directory.



* ``LANG``, ``LC_ALL``, and ``LANGUAGE`` are set to ``C``.



* ``TZ`` is set to ``GMT``.



* ``COLUMNS`` is set to ``80``.



* ``CDPATH`` and ``GREP_OPTIONS`` are set to an empty string.



Cram also provides the following environment variables to tests:



* ``CRAMTMP``, set to the test runner's temporary directory.



* ``TESTDIR``, set to the directory containing the test file.



.. _unified context diff: http://en.wikipedia.org/wiki/Diff#Unified_format

.. _its own example tests: http://bitbucket.org/brodie/cram/src/tip/examples/



News

----



Version 0.6

```````````

* Added support for specifying options in ``.cramrc`` (configurable

  with the ``CRAMRC`` environment variable).



* Added a ``--shell`` option to change the shell tests are run

  with. Contributed by `Kamil Kisiel`_.



* Added the long option ``--preserve-env`` for ``-E``.



.. _Kamil Kisiel: http://kamilkisiel.net/



Version 0.5 (Jan. 8, 2011)

``````````````````````````

* **The test format has changed:** Matching output not ending in a

  newline now requires the ``(no-eol)`` keyword instead of ending the

  line in ``%``.



* Matching output containing unprintable characters now requires the

  ``(esc)`` keyword. Real output containing unprintable characters

  will automatically receive ``(esc)``.



* If an expected line matches its real output line exactly, special

  matching like ``(re)`` or ``(glob)`` will be ignored.



* Regular expressions ending in a trailing backslash are now

  considered invalid.



* Added an ``--indent`` option for changing the default amount of

  indentation required to specify commands and output.



* Added support for specifying command line options in the ``CRAM``

  environment variable.



* The ``--quiet`` and ``--verbose`` options can now be used together.



* When running Cram under Python 3, Unicode-specific line break

  characters will no longer be parsed as newlines.



* Tests are no longer required to end in a trailing newline.



Version 0.4 (Sep. 28, 2010)

```````````````````````````

* **The test format has changed:** Output lines containing regular

  expressions must now end in ``(re)`` or they'll be matched

  literally. Lines ending with keywords are matched literally first,

  however.



* Regular expressions are now matched from beginning to end. In other

  words ``\d (re)`` is matched as ``^\d$``.



* In addition to ``(re)``, ``(glob)`` has been added. It supports

  ``*``, ``?``, and escaping both characters (and backslashes) using

  ``\``.



* **Environment settings have changed:** The ``-D`` flag has been

  removed, ``$TESTDIR`` is now set to the directory containing the

  ``.t`` file, and ``$CRAMTMP`` is set to the test runner's temporary

  directory.



* ``-i``/``--interactive`` now requires ``patch(1)``. Instead of

  ``.err`` files replacing ``.t`` files during merges, diffs are

  applied using ``patch(1)``. This prevents matching regular

  expressions and globs from getting clobbered.



* Previous ``.err`` files are now removed when tests pass.



* Cram now exits with return code ``1`` if any tests failed.



* If a test exits with return code ``80``, it's considered a skipped a

  test. This is useful for intentionally disabling tests when they

  only work on certain platforms or in certain settings.



* The number of tests, the number of skipped tests, and the number of

  failed tests are now printed after all tests are finished.



* Added ``-q``/``--quiet`` to suppress diff output.



* Added `contrib/cram.vim`_ syntax file for Vim. Contributed by `Steve

  Losh`_.



.. _contrib/cram.vim: http://bitbucket.org/brodie/cram/src/tip/contrib/cram.vim

.. _Steve Losh: http://stevelosh.com/



Version 0.3 (Sep. 20, 2010)

```````````````````````````

* Implemented resetting of common environment variables. This behavior

  can be disabled using the ``-E`` flag.



* Changed the test runner to first make its own overall random

  temporary directory, make ``tmp`` inside of it and set ``TMPDIR``,

  etc. to its path, and run each test with a random temporary working

  directory inside of that.



* Added ``--keep-tmpdir``. Temporary directories are named by test

  filename (along with a random string).



* Added ``-i``/``--interactive`` to merge actual output back to into

  tests interactively.



* Added ability to match command output not ending in a newline by

  suffixing output in the test with ``%``.



Version 0.2 (Sep. 19, 2010)

```````````````````````````

* Changed the test runner to run tests with a random temporary working

  directory.



Version 0.1 (Sep. 19, 2010)

```````````````````````````

* Initial release.



Development

-----------



Download the official development repository using Mercurial_::



    hg clone http://bitbucket.org/brodie/cram



Test Cram using Cram::



    make tests



Get a test coverage report using coverage.py_::



    make coverage



Visit Bitbucket_ if you'd like to fork the project, watch for new

changes, or report issues.



.. _Mercurial: http://mercurial.selenic.com/

.. _coverage.py: http://nedbatchelder.com/code/coverage/

.. _Bitbucket: http://bitbucket.org/brodie/cram