Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

https://github.com/parttimenerd/temci

An advanced benchmarking tool
https://github.com/parttimenerd/temci

Last synced: 25 days ago
JSON representation

An advanced benchmarking tool

Lists

README 

        
.. title:: temci



temci

=====



.. image:: https://github.com/parttimenerd/temci/actions/workflows/test.yml/badge.svg

    :target: https://github.com/parttimenerd/temci/actions/workflows/test.yml



.. image:: https://readthedocs.org/projects/temci/badge/?version=latest&style=plain

    :target: https://temci.readthedocs.org



An advanced benchmarking tool written in Python 3 that supports

`setting up an environment for benchmarking `_

and the generation of `visually appealing reports `_.



It runs on Linux systems and (rudimentarily) on macOS.



Why should you use temci?

-------------------------



temci allows you to easily measure the execution time (and other things)

of programs and compare them against each other resulting in a pretty

HTML5 based report. Furthermore it can set up the environment to ensure

benchmarking results with a low variance. The latter feature can be used

without using temci for benchmarking

by using `temci short shell `_.



Usage

-----



The main commands of temci are `temci exec `_ and

`temci report `_.



Suppose you want to see whether grepping for the strings that consist of ``a`` and ``b`` in the current

folder is slower than for strings that consist only of ``a``.



First we have to install temci (using `Nix `_, see below for more instructions):



.. code:: sh



    nix-env -f https://github.com/parttimenerd/temci/archive/master.tar.gz -i



After this, we can benchmark both commands with temci:



.. code:: sh



    # benchmark both commands 20 times

    temci short exec "grep '[ab]*' -R ." "grep 'a*' -R ." --runs 10



    # append --watch to get report (in which you can move with the arrow keys and scroll)

    # after every benchmark completed (use --watch_every to decrease interval)

    temci short exec "grep '[ab]*' -R ." "grep 'a*' -R ." --runs 10 --watch



    # if you want to improve the stability your benchmarks, run them with root privileges

    # the benchmarked programs are run with your current privileges

    temci short exec "grep '[ab]*' -R ." "grep 'a*' -R ." --runs 10 --sudo --preset usable



This results in a ``run_output.yaml`` file that should look like:



.. code:: yaml



    - attributes: {description: 'grep ''[ab]*'' -R .'}

      data:

        etime: [0.03, 0.02, 0.02, 0.03, 0.03, 0.03, 0.02, 0.03, 0.03, 0.02]

        … # other properties

    - attributes: {description: grep 'a*' -R .}

      data:

        etime: [0.02, 0.03, 0.02, 0.03, 0.03, 0.02, 0.03, 0.03, 0.02, 0.02]

        … # other properties

    - property_descriptions: {etime: elapsed real (wall clock) time, … }



For more information on the support measurement tools (like

`perf stat `_ and

`rusage `_),

the supported `plugins for setting up the environment `_

and more, see `temci exec `_.



We can now create a report from these benchmarking results using

`temci report `_.

We use the option ``--properties`` to include only the elapsed time in the

report to keep the report simple:



.. code:: sh



    > temci report run_output.yaml --properties etime

    Report for single runs

    grep '[ab]*' -R .    (   10 single benchmarks)

         etime mean =     2(6).(000)m, deviation = 18.84223%



    grep 'a*' -R .       (   10 single benchmarks)

         etime mean =     2(5).(000)m, deviation = 20.00000%



    Equal program blocks

         grep '[ab]*' -R .  ⟷  grep 'a*' -R .

             etime confidence =        67%, speed up =      3.85%



We see that there is no significant difference between the two commands.



There are multiple reporters besides the default

`console reporter `_.

Another reporter is the `html2 reporter `_

that produces an HTML report, use it by adding the ``--reporter html2`` option:



.. image:: http://mostlynerdless.de/files/report_readme/html_report.png

    :target: http://mostlynerdless.de/files/report_readme/report.html



Installation

------------



The simplest way is to use the `Nix package manager `_, after installing Nix, run:



.. code:: sh



          nix-env -f https://github.com/parttimenerd/temci/archive/master.tar.gz -i



Using pip requiring at least Python 3.6:



.. code:: sh



        sudo pip3 install temci



For more information see the Installation_ page.



Auto completion

~~~~~~~~~~~~~~~



Temci can generate auto completion files for bash and zsh. Add the following line to your `.bashrc` or `.zshrc`:



.. code:: sh



    . `temci_completion $0`



Using temci to set up a benchmarking environment

------------------------------------------------

Use the ``temci short shell COMMAND`` to run a command (``sh`` by default) in a shell that is inside

the benchmarking environment. Most options of ``temci short exec`` are supported.

For more information, see `temci shell `_.



Why is temci called temci?

--------------------------



The problem in naming programs is that most good program names are

already taken. A good program or project name has (in my opinion) the

following properties:



* it shouldn't be used on the relevant platforms (in this case: github and pypi)

* it should be short (no one wants to type long program names)

* it should be pronounceable

* it should have at least something to do with the program



temci is such a name. It's lojban for time (i.e. the time duration between two moments or events).



Contributing

------------



`Bug reports `_ and

`code contributions `_ are highly appreciated.



For more information, see the `Contributing `_ page.



.. _Installation: https://temci.readthedocs.io/en/latest/installation.html



.. _Resources: https://temci.readthedocs.io/en/latest/resources.html