Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/Vindaar/shell

A Nim mini DSL to execute shell commands
https://github.com/Vindaar/shell

Last synced: about 2 months ago
JSON representation

A Nim mini DSL to execute shell commands

Awesome Lists containing this project

README 

        
* shell

[[https://travis-ci.org/Vindaar/shell][https://travis-ci.org/Vindaar/shell.svg?branch=master]]



A mini Nim DSL to execute shell commands more conveniently.



** Usage

With this macro you can simply write

#+BEGIN_SRC nim

shell:

  touch foo

  mv foo bar

  rm bar

#+END_SRC

which is then rewritten to something equivalent to:

#+BEGIN_SRC nim

execShell("touch foo")

execShell("mv foo bar")

execShell("rm bar")

#+END_SRC

where =execShell= is a proc around =startProcess= for normal

compilation and =gorgeEx= when using NimScript.



Note: When using =NimScript= the given command is prepended by

#+BEGIN_SRC

&"cd {getCurrentDir()} && "

#+END_SRC

in order to switch the evaluation into the directory of the =shell=

call.

The same is achieved on the compiled backend by the =poEvalCommand=

argument to =startProcess=.



See [[Full expansion of the macro]] below for more details and how to read

the exit code of executed commands.



Most simple things should work as expected. See below for some known

quirks.



** ~one~ and ~pipe~



By default each line in the =shell= macro will be handled by a

different call to =execShell=. If you need several commands, which

depend on the state of the previous, you may do so via the =one=

command like so:

#+BEGIN_SRC nim

shell:

  one:

    mkdir foo

    cd foo

    touch bar

    cd ".."

    rm foo/bar

#+END_SRC



Similar to the =one= command, the =pipe= command exists. This concats

the command via the shell pipe =|=:

#+BEGIN_SRC nim

shell:

  pipe:

    cat test.txt

    head -3

#+END_SRC

will produce:

#+BEGIN_SRC nim

execShell("cat test.txt | head -3")

#+END_SRC



Both of these can even be combined!

#+BEGIN_SRC nim

shell:

  one:

    mkdir foo

    pushd foo

    echo "Hallo\nWorld" > test.txt

    pipe:

      cat test.txt

      grep H

    popd

    rm foo/test.txt

    rmdir foo

#+END_SRC

will work just as expected, echoing =Hallo= in the shell.



** Handling programs that require user input



Some terminal commands will require user input. Starting from version

=v0.5.0=, basic support for a =expect= / =send= feature is

available. It allows for functionality similar to the

[[https://linux.die.net/man/1/expect][=expect(1)= program]].



Let's consider two simple examples. Assuming we have a script that

reads from =stdin= such as:



=tExpect.sh=:

#+begin_src sh

#!/bin/sh



echo "Hello world. Your name?"

read foo

echo "Your name is" $foo

#+end_src



calling this script using the =shell= macro would normally

#+begin_src nim

import shell

shell:

  ./tExpect.sh

#+end_src

cause the nim program to simply stop upon encountering the =read=

line. 



Now we can do just this: 

#+begin_src nim

shell:

  ./tExpect.sh

  send: "BaFu"

#+end_src



This goes full rogue mode and just sends "BaFu", regardless what the question is. 



You can also specifiy what answer you send to which question.

By using =expect= / =send= we can handle it this way:



#+begin_src nim

import shell

shell:

  ./tExpect.sh

  expect: "Your name?"

  send: "BaFu"

#+end_src

As you notice the =expect= line only contains the second part of the

last printed line of the shell script. That is because the =expect=

functionality tries to match the shell output line by line using one

of the following:

- the =expect= line matches exactly

- the line starts with the =expect= line

- the line ends with the =expect= line (useful for long outputs that

  end with ="y/N"= like constructs)

(this may become configurable in the future)



Another example would be installing a nimble package and dealing with

the possibility of having to upgrade / overwrite package:

#+begin_src nim

import shell

shell:

  nimble install foo

  expect: "Overwrite? [y/N]"

  send: "y"

#+end_src

which handles such situations programatically.



*Note*: You may have multiple =expect= / =send= constructs in a single

=shell= call. But keep in mind that every =expect= must have a =send=.



** Nim symbol quoting



*NOTE:* In a previous version this was done via accented quotes

=`=. For the old behavior compile with =-d:oldQuote=.



Another important feature to make this library useful is quoting of

Nim symbols.



This is handled via parenthesis =()= (if you need to run something in

a subshell unfortunately that will have to be done with an explicit

string now). Any tree in =()= is subject to quoting. That means if an

identifier within =()= is preceded by a =$=, the symbol is

unquoted. Note however that for the moment only a single variable may

be quoted in each =()=.



The simplest case would be:

#+BEGIN_SRC nim

let name = "Vindaar"

shell:

  echo Hello from ($name)

#+END_SRC

which will perform the call:

#+BEGIN_SRC nim

execShell(&"echo Hello from {name}!")

#+END_SRC

and after the call to =strformat.&=:

#+BEGIN_SRC nim

execShell("echo Hello from Vindaar!")

#+END_SRC



*** Appending to a Nim identifier



Assuming we have a filename identifier and we want to convert some

image from =png= to =jpg= with image magick. The simplest command

should look like:

#+BEGIN_SRC sh

convert myimage.png myimage.jpg

#+END_SRC



This can be done in several ways.



**** Using dot expressions and no string literals:

#+BEGIN_SRC nim

let fname = "myimage"

shell:

  convert ($fname).png ($fname).jpg

#+END_SRC

Note that this is a special case. Continuing after a =()= quote

without literal strings will only work for dot expressions. For

instance:

#+BEGIN_SRC nim

let fname = "myimage"

shell:

  convert ($fname)".png" ($fname)".jpg"

#+END_SRC

will wrongly be converted to:

#+BEGIN_SRC sh

convert myimage .png myimage .jpg

#+END_SRC

which is obviously not what one would expect.



**** Using string literals:

#+BEGIN_SRC nim

let fname = "myimage"

shell:

  convert ($fname".pdf") ($fname".png")

#+END_SRC

In contrast to the wrong example shown above, this will work as

expected.



This is especially useful for cases without dot expressions after the

quoted nim identifier.



*** Appending a Nim identifier to a string literal



The other example would be appending a Nim identifier to a literal

string. For instance in case we have a filename, which we create at

run time and we wish to hand it to some command which takes an

argument, which is must be given without a space like:

#+BEGIN_SRC sh

./myBin input --out=output

#+END_SRC



In this case one of the following ways works:



**** using =()= after a string literal:

#+BEGIN_SRC nim

let outfile = "myoutput.txt"

shell:

  ./myBin input "--out="($outfile)

#+END_SRC

If the =()= appears after the literal we can correctly generate the

string without a space (in comparison to the case presented above when

a string literal follows a =()=).



**** For more predictable behavior, put the string literal also into

   =()=:

#+BEGIN_SRC nim

let outfile = "myoutput.txt"

shell:

  ./myBin input ("--out="$outfile)

#+END_SRC



*** General remark on predictability



*NOTE:* previously this section said to handle quoting + concatenation

with strings both in the case of with and without space with =()= for

the most predictable behavior. But that was a bad idea from my side!

If you need spaces, simply put it outside the =()= and use a space!



The =doAssert= below is to be understood in the context of the =shell=

macro. To summarize the above then:

#+BEGIN_SRC nim

let outfile = "myoutput.txt"

doAssert ("--out="$outfile) == &"--out={outfile}" # <- without space, ident after

doAssert "--out" ($outfile) == &"--out {outfile}" # <- with space, ident after

let fname = "myimage"

doAssert ($outfile".jpg") == &"{fname}.jpg" # <- without space, ident first

doAssert ($outfile) "image2" == &"{outfile} image2" # <- without space, ident first

#+END_SRC



*NOTE 2:* For the moment however, the =()= usage is restricted to a

single string literal (or something that is convertible to a string

via the =stringify= proc) and a single Nim identifier! This

restriction will maybe be removed in the future.



This syntax also works for more complicated Nim expressions than a

simple identifier:

#+BEGIN_SRC nim

const t = (a: "name", b: 5.5)

doAssert ("--out="$(t.a))

doAssert ("--out="$t.a)

#+END_SRC

both work. Of course =t= needn't be a tuple. It can also be an object

or even a function call, like for instance extracting a filename

within a call:

#+BEGIN_SRC nim

import os, shell

let path = "/some/user/path/toAFile.txt"

shell:

  ./myBin ("--inputFile="$(path.extractFilename))

#+END_SRC

should produce:

#+BEGIN_SRC sh

./myBin --inputFile=toAFile.txt

#+END_SRC



** Accented quotes



*NOTE*: In a previous version accented quotes were also used to quote

Nim identifiers. That use case is now handled via parentheses. For the

old behavior compile with =-d:oldQuote=.



Accented quotes allow you to hand raw strings.



Note: this has the downside of disallowing =`= as a token to be handed

to the shell. If you want to use the shell's =`=, you need to put the

appropriate command into quotation marks.



*** Raw strings

If you want to hand a literal string to the shell, you may do so by

putting it into accented quotes:

#+BEGIN_SRC nim

echo `hello`

#+END_SRC

will be rewritten to

#+BEGIN_SRC nim

execShell("echo \"hello\"")

#+END_SRC



For a string consisting of multiple commands / words, put quotation

marks around it:

#+BEGIN_SRC sh

echo `"Hello from Nim!"`

#+END_SRC

which will then also be rewritten to:

#+BEGIN_SRC nim

execShell("echo \"Hello from Nim!\"")

#+END_SRC



** Assignment of results to Nim variables



Also useful is assignment of the result of a shell call to a Nim

string. This can be done with the =shellAssign= macro. It is a little

special compared to the =shell= and =shellEcho= macros. It only

supports a single statement (*), which needs to be an assignment of a

shell call of the syntax presented above to a Nim variable, such as:

#+BEGIN_SRC nim

var name = ""

shellAssign:

  name = echo Araq

assert name == "Araq"

#+END_SRC

Here the left =name= is the Nim variable (note: this is an exception

of the Nim symbol quoting mentioned above!), whereas the right hand

side is an arbitrary shell call, in this case a simple call to

=echo=. The Nim variable will be assigned the result of the shell

call, by being rewritten to:

#+BEGIN_SRC nim

var name = ""

name = asgnShell("echo Araq")

assert name == "Araq"

#+END_SRC

=asgnShell= is internally called by =execShell= mentioned

above. =asgnShell= itself performs the calls to =execCmdEx= (or =exec=

for NimScript).



(*): a single statement is not entirely precise, because the =one= and

=pipe= operators can be used in combination with the assignment! For

example the following is also possible:

#+BEGIN_SRC nim

var res = ""

shellAssign:

  res = pipe:

    seq 0 1 10

    tail -3

assert res == "8\n9\n10"

#+END_SRC



** NimScript & Nim compile time (~nimvm~)



This macro can also be used in NimScript as well as at regular Nim

compile time (i.e. in ~nimvm~ contexts). It uses ~gorgeEx~ in this

case. Some features may not work perfectly. Instead of using the

current working directory, it always starts at the path of the current

project being compiled! So you may need to prepend your own ~cd ~

depending on where you need to run your commands.



** Known issues



Certain things unfortunately *have* to go into quotation marks. As

seen in the =one= example above, the simple =..= is not allowed.



Variable assignments in the shell need to be handed via a string

literal:

#+BEGIN_SRC nim

shell:

  one:

    "a=`echo hello`"

    echo $a

#+END_SRC



Also if you need assignment via `:`, you need to also put it into quotation

marks, as the Nim tree is not unambiguous enough to properly parse the

resulting command. Say you wish to compile a Nim program, you might want to do:

#+BEGIN_SRC nim

shell:

  nim c "--out:noTest" test.nim

#+END_SRC



A slightly different case is assignment via `=`. A single `=`

assignment in a command is possible, but not more. That is due to a

Nim parser limitation (as you cannot have "nested" `nnkAsgn` nodes).



So this:

#+begin_src sh

shel:

  foo --bar --option=value --more

#+end_src

is fine, but this:

#+begin_src sh

shell:

  foo --bar --option=value --secondOption=value2

#+end_src

is *not* valid Nim syntax. Fortunately, in most cases the `=` sign is

optional anyway and you may just drop (one or both):

#+begin_src sh

shel:

  foo --bar --option value --secondOption value2

#+end_src

is valid and means the same for most programs.



In general, if in doubt you can just write strings or triple string

(to pass a ="= to the shell).



** Full expansion of the macro



As mentioned at the top of the README, the expansion shown is

simplified (as a matter of fact it was as simple once, but has since

become more complex).



The full expansion of the first example is:

#+BEGIN_SRC nim

discard block:

  var outputStr381052 = ""

  var exitCode381051: int

  if exitCode381051 ==

      0:

    let tmp381063 = execShell("touch foo")

    outputStr381052 = outputStr381052 &

        tmp381063[0]

    exitCode381051 = tmp381063[1]

  else:

    echo "Skipped command `" & "touch foo" &

        "` due to failure in previous command!"

  if exitCode381051 ==

      0:

    let tmp381064 = execShell("mv foo bar")

    outputStr381052 = outputStr381052 &

        tmp381064[0]

    exitCode381051 = tmp381064[1]

  else:

    echo "Skipped command `" & "mv foo bar" &

        "` due to failure in previous command!"

  if exitCode381051 ==

      0:

    let tmp381065 = execShell("rm bar")

    outputStr381052 = outputStr381052 &

        tmp381065[0]

    exitCode381051 = tmp381065[1]

  else:

    echo "Skipped command `" & "rm bar" &

        "` due to failure in previous command!"

  (outputStr381052, exitCode381051)

#+END_SRC



As can be seen from the expansion above, successive commands are only

run, if the exit code of the previous command was 0, while the output

is appended to the previous command's output.



The normal =shell= command discards the return value of the block. If

you want to keep it, use the =shellVerbose= macro:

#+BEGIN_SRC nim

let res = shellVerbose:

  someCommand

#+END_SRC

where =res= will be of type =tuple[output: string, exitCode: string]=

according to the expansion above.



** Debugging

In order to see what's going on, you can either compile your program

with the =-d:debugShell= flag, which will then echo the rewritten

commands during compilation.

Alternatively in order to avoid calling the commands immediately, you

may use the =shellEcho= macro instead. It simply echoes the commands

that would otherwise be run.



** Error reporting



By default ~shell~ prints output messages to stdout:



#+BEGIN_SRC nim :exports both :results scalar

import shell



shell:

  ls

#+END_SRC



#+RESULTS:

: shellCmd: ls

: shell> nim.cfg

: shell> README.org

: shell> shell

: shell> shell.nim

: shell> shell.nim.bin

: shell> shell.nimble

: shell> tests



What is printed to stdout can be configured by using defines:



- ~shellNoDebugOutput~ :: Do not print command output

- ~shellNoDebugError~ :: Do not print error output

- ~shellNoDebugCommand~ :: Do not print command being executed

- ~shellNoDebugRuntime~ :: When error occurs do not print failed command



By default these are disabled - to enable use either

~-d:shellNoDebug*~ or use the ~{.define(shellNoDebug*).}~ pragma



#+BEGIN_SRC nim :exports both :results scalar

{.define(shellNoDebugOutput).}



import shell



shell:

  ls

#+END_SRC



#+RESULTS:

: shellCmd: ls



The default ~shellVerbose~ command combines stderr and stdout into

single result. To get =stdout=, =stderr= and the return code

separately use ~shellVerboseErr~. Both of these templates have an

overload that takes ~set[DebugOutputKind]~ to control printing

settings:



#+BEGIN_SRC nim :exports both :results scalar

import shell



let (res, err, code) = shellVerboseErr {dokCommand}:

  echo "test"



echo "Returned string: '", res, "' with exit code ", code



#+END_SRC



#+RESULTS:

: shellCmd: echo test

: Returned string: 'test' with exit code 0



Printing errors directly into stdout is good solution for most of the

use cases, but sometimes it is necessary to provide more sophisticated

error handing - throwing an exception when the command failed. To

switch to exceptions use ~-d:shellThrowException~. It will

automatically disable all other output types in the default

configuration.



#+begin_src nim :exports both :results scalar

{.define(shellThrowException).}



import shell, strutils



try:

  shell:

    ls -l

    ls -z

except ShellExecError:

  let e = cast[ShellExecError](getCurrentException())

  echo e.msg # Error message describing what happened

  echo "command was: ", e.cmd # Original command string

  assert e.cmd == "ls -z"

  echo "return code: ", e.retcode # Return code

  echo "regular out: ", e.outstr # Stdout from command

  echo "error outpt: "

  for l in e.errstr.split('\n'): # Stderr from the command

    echo "  ", l

#+end_src



#+RESULTS:

: Command ls -z exited with non-zero code

: command was: ls -z

: return code: 2

: regular out:

: error outpt:

:   ls: invalid option -- 'z'

:   Try 'ls --help' for more information.



On command failure ~ShellExecError~ is raised.



Note that some commands output error messages into ~stdout~ rather

than into ~stderr~ - it might be necessary to check both. In this

particular example content of the ~stderr~ is largely meaningless:

actual reason for error was printed into ~stdout~.



#+begin_src nim :exports both :results scalar

{.define(shellThrowException).}



import shell, strutils



try:

  shell:

    ngspice -b "/tmp/ngpsice-simulation/zzz.netkRs8jE"

except ShellExecError:

  let e = cast[ShellExecError](getCurrentException())

  echo e.msg # Error message describing what happened

  echo "command was: ", e.cmd # Original command string

  echo "exec direct: ", e.cwd # 

  echo "return code: ", e.retcode # Return code

  echo "regular out: \n====\n", e.outstr # Stdout from command

  echo "====\nerror outpt: \n====\n", e.errstr # Stderr from the command

#+end_src



#+RESULTS:

#+begin_example

Command ngspice -b /tmp/ngpsice-simulation/zzz.netkRs8jE exited with non-zero code

command was: ngspice -b /tmp/ngpsice-simulation/zzz.netkRs8jE

exec direct: /home/test/workspace/git-sandbox/shell

return code: 1

regular out: 

====



====

error outpt: 

====

/tmp/ngpsice-simulation/zzz.netkRs8jE: No such file or directory

#+end_example