https://github.com/nightmachinery/brish

Safely embed Zsh in Python.
https://github.com/nightmachinery/brish

bridge exec execute executor interop interoperability interpreters modern-shell process-manager python shell subprocess xonsh zsh

Last synced: 7 days ago
JSON representation

Safely embed Zsh in Python.

• Host: GitHub
• URL: https://github.com/nightmachinery/brish
• Owner: NightMachinery
• Created: 2020-04-13T12:41:52.000Z (over 4 years ago)
• Default Branch: master
• Last Pushed: 2024-10-14T11:24:15.000Z (24 days ago)
• Last Synced: 2024-10-27T12:45:18.776Z (10 days ago)
• Topics: bridge, exec, execute, executor, interop, interoperability, interpreters, modern-shell, process-manager, python, shell, subprocess, xonsh, zsh
• Language: Python
• Homepage:
• Size: 109 KB
• Stars: 12
• Watchers: 3
• Forks: 0
• Open Issues: 8
• Metadata Files:
  • Readme: readme.org

Awesome Lists containing this project

README

        
#+TITLE: Brish

#+begin_html

























#+end_html

* Guide

** Installation

~pip install -U brish~

Or install the latest master (recommended, as I might have forgotten to push a new versioned update):

~pip install git+https://github.com/NightMachinary/brish~

You need a recent Python version, as Brish uses some of the newer metaprogramming APIs. Obviously, you also need zsh installed.

** Quickstart

#+begin_src python :session p1 :results silent :tangle tests/test_tangled1.py

from brish import z, zp, Brish

#+end_src

#+begin_src python :session p1 :results silent :exports none :tangle tests/test_tangled1.py

NI = True

#+end_src

#+begin_src python :session p1 :results silent :exports none

NI = False

#+end_src

#+name: t1

#+begin_src python :session p1 :results value :exports both :tangle tests/test_tangled1.py

name="A$ron"

z("echo Hello {name}")

#+end_src

#+RESULTS: t1

#+begin_example

Hello A$ron

#+end_example

#+begin_src python :session p1 :var t1=t1 :results value :exports none :tangle tests/test_tangled1.py

def test1():

    assert t1 == "Hello A$ron"

    return True

NI or test1()

#+end_src

#+RESULTS:

#+begin_example

True

#+end_example

~z~ automatically converts Python lists to shell lists:

#+name: t2

#+begin_src python :session p1 :results value :exports both :tangle tests/test_tangled1.py

alist = ["# Fruits", "1. Orange", "2. Rambutan", "3. Strawberry"]

z("for i in {alist} ; do echo $i ; done")

#+end_src

#+RESULTS: t2

#+begin_example

# Fruits

1. Orange

2. Rambutan

3. Strawberry

#+end_example

#+begin_src python :session p1 :var t2=t2 :results value :exports none :tangle tests/test_tangled1.py

def test2():

    assert t2 == """# Fruits

1. Orange

2. Rambutan

3. Strawberry"""

NI or test2()

#+end_src

#+RESULTS:

#+begin_example

None

#+end_example

~z~ returns a ~CmdResult~ (more about which later):

#+begin_src python :session p1 :results value :exports both

res = z("date +%Y")

repr(res)

#+end_src

#+RESULTS:

#+begin_example

CmdResult(retcode=0, out='2021\n', err='', cmd=' date +%Y ', cmd_stdin='')

#+end_example

You can use ~zp~ as a shorthand for ~print(z(...).outerr, end='')~:

#+begin_src python :session p1 :results output :exports both

for i in range(10):

    cmd = "(( {i} % 2 == 0 )) && echo {i} || {{ echo Bad Odds'!' >&2 }}" # Using {{ and }} as escapes for { and }

    zp(cmd)

    print(f"Same thing: {z(cmd).outerr}", end='')

#+end_src

#+RESULTS:

#+begin_example

0

Same thing: 0

Bad Odds!

Same thing: Bad Odds!

2

Same thing: 2

Bad Odds!

Same thing: Bad Odds!

4

Same thing: 4

Bad Odds!

Same thing: Bad Odds!

6

Same thing: 6

Bad Odds!

Same thing: Bad Odds!

8

Same thing: 8

Bad Odds!

Same thing: Bad Odds!

#+end_example

~CmdResult~ is true if its return code is zero:

#+name: t3

#+begin_src python :session p1 :results output :exports both :tangle tests/test_tangled1.py

if z("test -e ~/"):

    print("HOME exists!")

else:

    print("We're homeless :(")

#+end_src

#+RESULTS: t3

#+begin_example

HOME exists!

#+end_example

#+begin_src python :session p1 :var t3=t3 :results value :exports none :tangle tests/test_tangled1.py

assert t3 == "HOME exists!"

#+end_src

#+RESULTS:

~CmdResult~ is smart about iterating:

#+name: t4

#+begin_src python :session p1 :results output :exports both :tangle tests/test_tangled1.py

for path in z("command ls ~/tmp/"): # `command` helps bypass potential aliases defined on `ls`

    zp("du -h ~/tmp/{path}") # zp prints the result

#+end_src

#+RESULTS: t4

#+begin_example

524K	/Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/c01ed1a32d65c8d4ecb9095509e61f97

524K	/Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/36b77e6b3b7fde31f2fc4f182c0ecf82

1.3M	/Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/tumblr/dreamcorp420

1.3M	/Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/tumblr

  0B	/Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/34cc02221710caf309bff5ca96808d7a

520K	/Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/6cc3d153426e2b6d1ac0f3736aaf74a1

2.9M	/Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43

  0B	/Users/evar/tmp/8826215ac0ed61d617906f658322fce7

348K	/Users/evar/tmp/IMG_0396.PNG

 44K	/Users/evar/tmp/a2.jpg

 40K	/Users/evar/tmp/a4.jpg

8.0K	/Users/evar/tmp/bills

  0B	/Users/evar/tmp/garden

468K	/Users/evar/tmp/image-14000213234237913.png

 40K	/Users/evar/tmp/photo_2021-05-08_00-35-24.jpg

152K	/Users/evar/tmp/photo_2021-05-08_00-55-29.jpg

8.0K	/Users/evar/tmp/tumblr

4.0M	/Users/evar/tmp/tumblr_2c0ad7a3fba563996c9abaedc5e8d4f7_356ef3d9_1280.gif

576K	/Users/evar/tmp/tumblr_5a2868650b058c42a7d141b8a2f474bc_eac04dc0_1280.jpg

976K	/Users/evar/tmp/tumblr_5cc2e0e48418ec3c9eb200d151daf647_e44e419b_1280.jpg

 44K	/Users/evar/tmp/tumblr_6c90d77a676cf20fc096cc19220af4ab_e124dbec_540.gif.mp4

  0B	/Users/evar/tmp/tumblr_70675efa5303a58292957ac942663309_f48499c2_1280.jpg

4.0K	/Users/evar/tmp/tumblr_70675efa5303a58292957ac942663309_f48499c2_1280.jpg.aria2

656K	/Users/evar/tmp/tumblr_bc2259b471f792065eb6707b7c29d27e_97f97d94_1280.jpg

1.0M	/Users/evar/tmp/tumblr_ec36fde70ee0264cdc2f61f394181c61_575227e7_1280.jpg

 84K	/Users/evar/tmp/view.php

#+end_example

#+begin_src python :session p1 :results value :exports both

res = z("""echo This is stdout

           echo This is stderr >&2

           (exit 6) # this is the return code""")

repr(res.out)

#+end_src

#+RESULTS:

#+begin_example

This is stdout\n

#+end_example

~CmdResult.outrs~ strips the final newlines:

#+begin_src python :session p1 :results value :exports both

repr(res.outrs)

#+end_src

#+RESULTS:

#+begin_example

This is stdout

#+end_example

#+begin_src python :session p1 :results value :exports both

repr(res.err)

#+end_src

#+RESULTS:

#+begin_example

This is stderr\n

#+end_example

#+begin_src python :session p1 :results value :exports both

res.retcode

#+end_src

#+RESULTS:

#+begin_example

6

#+end_example

#+begin_src python :session p1 :results value :exports both

res.longstr

#+end_src

#+RESULTS:

#+begin_example

cmd:  echo This is stdout

           echo This is stderr >&2

           (exit 6) # this is the return code

stdout:

This is stdout

stderr:

This is stderr

return code: 6

#+end_example

By default, ~z~ doesn't fork. So we can use it to change the state of the running zsh session:

#+begin_src python :session p1 :results value :exports both

z("""

(($+commands[imdbpy])) || pip install -U imdbpy

imdb() imdbpy search movie --first "$*"

""")

z("imdb Into the Woods 2014")

#+end_src

#+RESULTS:

#+begin_example

Movie

=====

Title: Into the Woods (2014)

Genres: Adventure, Comedy, Drama, Fantasy, Musical.

Director: Rob Marshall.

Writer: James Lapine, James Lapine.

Cast: Anna Kendrick (Cinderella), Daniel Huttlestone (Jack), James Corden (Baker / Narrator), Emily Blunt (Baker's Wife), Christine Baranski (Stepmother).

Runtime: 125.

Country: United States.

Language: English.

Rating: 5.9 (134093 votes).

Plot: A witch tasks a childless baker and his wife with procuring magical items from classic fairy tales to reverse the curse put on their family tree.

#+end_example

We can force a fork. This is useful to make your scripts more robust.

#+begin_src python :session p1 :results output :exports both

print(z("exit 7", fork=True).retcode)

zp("echo 'Still alive!'")

#+end_src

#+RESULTS:

#+begin_example

7

Still alive!

#+end_example

Working with stdin:

#+begin_src python :session p1 :results value :exports both

# the intuitive way

a="""1

2

3

4

5

"""

z("<<<{a} wc -l")

#+end_src

#+RESULTS:

#+begin_example

6

#+end_example

#+begin_src python :session p1 :results value :exports both

z("wc -l", cmd_stdin=a)

#+end_src

#+RESULTS:

#+begin_example

5

#+end_example

** More Details

The stdin will by default be set to the empty string:

#+begin_src python :session p1 :results output :exports both

zp("cat")

zp("echo 'As you see, the previous command produced no output. It also did not block.'")

#+end_src

#+RESULTS:

#+begin_example

as you see, the previous command produced no output. It also did not block.

#+end_example

~z~ escapes your Python variables automagically:

#+begin_src python :session p1 :results value :exports both

python_var = "$HOME"

z("echo {python_var}")

#+end_src

#+RESULTS:

#+begin_example

$HOME

#+end_example

Turning off the auto-escape:

#+begin_src python :session p1 :results value :exports both

z("echo {python_var:e}")

#+end_src

#+RESULTS:

#+begin_example

/Users/evar

#+end_example

Working with Python bools from the shell:

#+begin_src python :session p1 :results value :exports both

z("test -n {True:bool}").retcode

#+end_src

#+RESULTS:

#+begin_example

0

#+end_example

#+begin_src python :session p1 :results value :exports both

z("test -n {False:bool}").retcode

#+end_src

#+RESULTS:

#+begin_example

1

#+end_example

Working with NUL-terminated output:

#+begin_src python :session p1 :results output :exports both

for f in z("fd -0 . ~/tmp").iter0():

    zp("echo {f}")

#+end_src

#+RESULTS:

#+begin_example

/Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43

/Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/34cc02221710caf309bff5ca96808d7a

/Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/36b77e6b3b7fde31f2fc4f182c0ecf82

/Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/36b77e6b3b7fde31f2fc4f182c0ecf82/tumblr_9527f4f6d2f1a39ef2b839780831f38f_859e5e2b_2048.jpg

/Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/36b77e6b3b7fde31f2fc4f182c0ecf82/tumblr_dd64a6ced93d19ffe78b47cf3439373d_e8e18fb0_2048.jpg

/Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/6cc3d153426e2b6d1ac0f3736aaf74a1

/Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/6cc3d153426e2b6d1ac0f3736aaf74a1/tumblr_9527f4f6d2f1a39ef2b839780831f38f_859e5e2b_2048.jpg

/Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/6cc3d153426e2b6d1ac0f3736aaf74a1/tumblr_dd64a6ced93d19ffe78b47cf3439373d_e8e18fb0_2048.jpg

/Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/c01ed1a32d65c8d4ecb9095509e61f97

/Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/c01ed1a32d65c8d4ecb9095509e61f97/tumblr_9527f4f6d2f1a39ef2b839780831f38f_859e5e2b_2048.jpg

/Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/c01ed1a32d65c8d4ecb9095509e61f97/tumblr_dd64a6ced93d19ffe78b47cf3439373d_e8e18fb0_2048.jpg

/Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/tumblr

/Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/tumblr/dreamcorp420

/Users/evar/tmp/83e93d36396014e0cd979ddcad2d9d43/tumblr/dreamcorp420/tumblr_dreamcorp420_650543836474589184_01.gif

/Users/evar/tmp/8826215ac0ed61d617906f658322fce7

/Users/evar/tmp/IMG_0396.PNG

/Users/evar/tmp/a2.jpg

/Users/evar/tmp/a4.jpg

/Users/evar/tmp/bills

/Users/evar/tmp/garden

/Users/evar/tmp/image-14000213234237913.png

/Users/evar/tmp/photo_2021-05-08_00-35-24.jpg

/Users/evar/tmp/photo_2021-05-08_00-55-29.jpg

/Users/evar/tmp/tumblr

/Users/evar/tmp/tumblr_2c0ad7a3fba563996c9abaedc5e8d4f7_356ef3d9_1280.gif

/Users/evar/tmp/tumblr_5a2868650b058c42a7d141b8a2f474bc_eac04dc0_1280.jpg

/Users/evar/tmp/tumblr_5cc2e0e48418ec3c9eb200d151daf647_e44e419b_1280.jpg

/Users/evar/tmp/tumblr_6c90d77a676cf20fc096cc19220af4ab_e124dbec_540.gif.mp4

/Users/evar/tmp/tumblr_70675efa5303a58292957ac942663309_f48499c2_1280.jpg

/Users/evar/tmp/tumblr_70675efa5303a58292957ac942663309_f48499c2_1280.jpg.aria2

/Users/evar/tmp/tumblr_bc2259b471f792065eb6707b7c29d27e_97f97d94_1280.jpg

/Users/evar/tmp/tumblr_ec36fde70ee0264cdc2f61f394181c61_575227e7_1280.jpg

/Users/evar/tmp/view.php

#+end_example

You can bypass the automatic iterable conversion by converting the iterable to a string first:

#+begin_src python :session p1 :results value :exports both

z("echo {'    '.join(map(str,alist))}")

#+end_src

#+RESULTS:

#+begin_example

# Fruits    1. Orange    2. Rambutan    3. Strawberry

#+end_example

Normal Python formatting syntax works as expected:

#+begin_src python :session p1 :results value :exports both

z("echo {67:f}")

#+end_src

#+RESULTS:

#+begin_example

67.0

#+end_example

#+begin_src python :session p1 :exports both :results verbatim

z("echo {[11, 45]!s}")

#+end_src

#+RESULTS:

#+begin_example

[11, 45]

#+end_example

You can obviously nest your ~z~ calls:

#+begin_src python :session p1 :results value :exports both

z("""echo monkey$'\n'{z("curl -s https://www.poemist.com/api/v1/randompoems | jq --raw-output '.[0].content'")}$'\n'end | sed -e 's/monkey/Random Poem:/'""")

#+end_src

#+RESULTS:

#+begin_example

Random Poem:

’Tis said that the Passion Flower,

   With its figures of spear and sword

And hammer and nails, is a symbol

   Of the Woe of our Blessed Lord.

So still in the Heart of Beauty

   Has been hidden, since Life drew breath,

The sword and the spear of Anguish,

   And the hammer and nails of Death.

end

#+end_example

*** The Brish Class

~z~ and ~zp~ are just convenience methods:

#+begin_example

bsh = Brish()

z = bsh.z

zp = bsh.zp

zq = bsh.zsh_quote

zs = bsh.zstring

#+end_example

You can use ~Brish~ instances yourself (all arguments to it are optional). The boot command ~boot_cmd~ allows you to easily initialize the zsh session:

#+begin_src python :session p1 :results value :exports both

my_own_brish = Brish(boot_cmd="mkdir -p ~/tmp ; cd ~/tmp")

my_own_brish.z("echo $PWD")

#+end_src

#+RESULTS:

#+begin_example

/Users/evar/tmp

#+end_example

~Brish.z~ itself is sugar around ~Brish.zstring~ and ~Brish.send_cmd~:

#+begin_src python :session p1 :results value :exports both

cmd_str = my_own_brish.zstring("echo zstring constructs the command string that will be sent to zsh. It interpolates the Pythonic variables: {python_var} {alist}")

cmd_str

#+end_src

#+RESULTS:

#+begin_example

 echo zstring constructs the command string that will be sent to zsh. It interpolates the Pythonic variables: '$HOME' '# Fruits' '1. Orange' '2. Rambutan' '3. Strawberry'

#+end_example

#+begin_src python :session p1 :results value :exports both

my_own_brish.send_cmd(cmd_str)

#+end_src

#+RESULTS:

#+begin_example

zstring constructs the command string that will be sent to zsh. It interpolates the Pythonic variables: $HOME # Fruits 1. Orange 2. Rambutan 3. Strawberry

#+end_example

You can restart a Brish instance:

#+begin_src python :session p1 :results output :exports both

my_own_brish.z("a=56")

my_own_brish.zp("echo Before restart: $a")

my_own_brish.restart()

my_own_brish.zp("echo After restart: $a")

my_own_brish.zp("echo But the boot_cmd has run in the restarted instance, too: $PWD")

#+end_src

#+RESULTS:

#+begin_example

Before restart: 56

After restart:

But the boot_cmd has run in the restarted instance, too: /Users/evar/tmp

#+end_example

~Brish~ is threadsafe. I have built [[https://github.com/NightMachinary/BrishGarden][BrishGarden]] on top of ~Brish~ to provide an HTTP REST API for executing zsh code (if wanted, in sessions). Using ~BrishGarden~, you can embed ~zsh~ in pretty much any programming language, and pay no cost whatsoever for its startup. It can also function as a remote code executor.

**** Parallel Execution Using =server_count=

=server_count= allows the underlying =zsh= instance of a =Brish= object to fork that many times, and so serve that many clients in parallel. This will not increase the startup time, as the forking happens after loading the =zsh= interpreter completely.

I have combined this with =GNU parallel= to easily parallelize my =zsh= functions.

#+begin_src python :session p1 :results output :exports both

n = 32

my_parallel_brish = Brish(server_count=n)

import logging

import threading

import time

def thread_function(name):

    logging.info("Thread %s: starting", name)

    my_parallel_brish.zp("echo Started {name} at $EPOCHREALTIME ; sleep 10 ; echo Finished {name} at $EPOCHREALTIME")

    logging.info("Thread %s: finishing", name)

if __name__ == "__main__":

    format = "%(asctime)s: %(message)s"

    logging.basicConfig(format=format, level=logging.INFO,

                        datefmt="%H:%M:%S")

    threads = list()

    now = float(z("echo $EPOCHREALTIME").outrs)

    for index in range(32):

        logging.info("Main    : create and start thread %d.", index)

        x = threading.Thread(target=thread_function, args=(index,))

        threads.append(x)

        x.start()

    for index, thread in enumerate(threads):

        logging.info("Main    : before joining thread %d.", index)

        thread.join()

        logging.info("Main    : thread %d done", index)

    end = float(z("echo $EPOCHREALTIME").outrs)

    print(f"Took {(end - now)}")

#+end_src

#+RESULTS:

#+begin_example

17:25:26: Main    : create and start thread 0.

17:25:26: Thread 0: starting

17:25:26: Main    : create and start thread 1.

17:25:26: Thread 1: starting

17:25:26: Main    : create and start thread 2.

17:25:26: Thread 2: starting

17:25:26: Main    : create and start thread 3.

17:25:26: Thread 3: starting

17:25:26: Main    : create and start thread 4.

17:25:26: Thread 4: starting

17:25:26: Main    : create and start thread 5.

17:25:26: Thread 5: starting

17:25:26: Main    : create and start thread 6.

17:25:26: Thread 6: starting

17:25:26: Main    : create and start thread 7.

17:25:26: Thread 7: starting

17:25:26: Main    : create and start thread 8.

17:25:26: Thread 8: starting

17:25:26: Main    : create and start thread 9.

17:25:26: Thread 9: starting

17:25:26: Main    : create and start thread 10.

17:25:26: Thread 10: starting

17:25:26: Main    : create and start thread 11.

17:25:26: Thread 11: starting

17:25:26: Main    : create and start thread 12.

17:25:26: Thread 12: starting

17:25:26: Main    : create and start thread 13.

17:25:26: Thread 13: starting

17:25:26: Main    : create and start thread 14.

17:25:26: Thread 14: starting

17:25:26: Main    : create and start thread 15.

17:25:26: Thread 15: starting

17:25:26: Main    : create and start thread 16.

17:25:26: Thread 16: starting

17:25:26: Main    : create and start thread 17.

17:25:26: Thread 17: starting

17:25:26: Main    : create and start thread 18.

17:25:26: Thread 18: starting

17:25:26: Main    : create and start thread 19.

17:25:26: Thread 19: starting

17:25:26: Main    : create and start thread 20.

17:25:26: Thread 20: starting

17:25:26: Main    : create and start thread 21.

17:25:26: Thread 21: starting

17:25:26: Main    : create and start thread 22.

17:25:26: Thread 22: starting

17:25:26: Main    : create and start thread 23.

17:25:26: Thread 23: starting

17:25:26: Main    : create and start thread 24.

17:25:26: Thread 24: starting

17:25:26: Main    : create and start thread 25.

17:25:26: Thread 25: starting

17:25:26: Main    : create and start thread 26.

17:25:26: Thread 26: starting

17:25:26: Main    : create and start thread 27.

17:25:26: Thread 27: starting

17:25:26: Main    : create and start thread 28.

17:25:26: Thread 28: starting

17:25:26: Main    : create and start thread 29.

17:25:26: Thread 29: starting

17:25:26: Main    : create and start thread 30.

17:25:26: Thread 30: starting

17:25:26: Main    : create and start thread 31.

17:25:26: Thread 31: starting

17:25:26: Main    : before joining thread 0.

Started 0 at 1620651326.2126729488

Finished 0 at 1620651336.2229239941

17:25:36: Thread 0: finishing

17:25:36: Main    : thread 0 done

17:25:36: Main    : before joining thread 1.

Started 1 at 1620651327.2022259235

Finished 1 at 1620651337.2120540142

17:25:37: Thread 1: finishing

17:25:37: Main    : thread 1 done

17:25:37: Main    : before joining thread 2.

Started 30 at 1620651327.2101778984

Finished 30 at 1620651337.2140960693

17:25:37: Thread 30: finishing

Started 2 at 1620651328.2068090439

Finished 2 at 1620651338.2182691097

17:25:38: Thread 2: finishing

17:25:38: Main    : thread 2 done

17:25:38: Main    : before joining thread 3.

Started 31 at 1620651328.2222359180

Finished 31 at 1620651338.2338199615

17:25:38: Thread 31: finishing

Started 7 at 1620651329.2063989639

Finished 7 at 1620651339.2115590572

17:25:39: Thread 7: finishing

Started 15 at 1620651330.2087130547

Finished 15 at 1620651340.2192440033

17:25:40: Thread 15: finishing

Started 21 at 1620651331.2160348892

Finished 21 at 1620651341.2246019840

17:25:41: Thread 21: finishing

Started 23 at 1620651332.2160398960

Finished 23 at 1620651342.2200219631

17:25:42: Thread 23: finishing

Started 9 at 1620651333.2236700058

Finished 9 at 1620651343.2359619141

17:25:43: Thread 9: finishing

Started 18 at 1620651334.2257950306

Finished 18 at 1620651344.2365601063

17:25:44: Thread 18: finishing

Started 12 at 1620651335.2241439819

Finished 12 at 1620651345.2335329056

17:25:45: Thread 12: finishing

Started 20 at 1620651336.2342200279

Finished 20 at 1620651346.2429049015

17:25:46: Thread 20: finishing

Started 16 at 1620651337.4859669209

Finished 16 at 1620651347.4899230003

17:25:47: Thread 16: finishing

Started 22 at 1620651338.2339038849

Finished 22 at 1620651348.2375440598

17:25:48: Thread 22: finishing

Started 19 at 1620651339.2459530830

Finished 19 at 1620651349.2504169941

17:25:49: Thread 19: finishing

Started 13 at 1620651340.2416980267

Finished 13 at 1620651350.2485001087

17:25:50: Thread 13: finishing

Started 10 at 1620651340.2490129471

Finished 10 at 1620651350.2568130493

17:25:50: Thread 10: finishing

Started 29 at 1620651341.2439520359

Finished 29 at 1620651351.2504179478

17:25:51: Thread 29: finishing

Started 25 at 1620651342.2465701103

Finished 25 at 1620651352.2498950958

17:25:52: Thread 25: finishing

Started 17 at 1620651343.2493131161

Finished 17 at 1620651353.2571830750

17:25:53: Thread 17: finishing

Started 28 at 1620651344.2550890446

Finished 28 at 1620651354.2586359978

17:25:54: Thread 28: finishing

Started 14 at 1620651345.2569661140

Finished 14 at 1620651355.2659308910

17:25:55: Thread 14: finishing

Started 5 at 1620651346.2559928894

Finished 5 at 1620651356.2631940842

17:25:56: Thread 5: finishing

Started 4 at 1620651347.2538421154

Finished 4 at 1620651357.2619009018

17:25:57: Thread 4: finishing

Started 3 at 1620651347.2638580799

Finished 3 at 1620651357.2686970234

17:25:57: Thread 3: finishing

17:25:57: Main    : thread 3 done

17:25:57: Main    : before joining thread 4.

17:25:57: Main    : thread 4 done

17:25:57: Main    : before joining thread 5.

17:25:57: Main    : thread 5 done

17:25:57: Main    : before joining thread 6.

Started 26 at 1620651348.2553079128

Finished 26 at 1620651358.2628009319

17:25:58: Thread 26: finishing

Started 27 at 1620651348.2706210613

Finished 27 at 1620651358.2781529427

17:25:58: Thread 27: finishing

Started 24 at 1620651349.2586579323

Finished 24 at 1620651359.2646100521

17:25:59: Thread 24: finishing

Started 11 at 1620651350.2648739815

Finished 11 at 1620651360.2702779770

17:26:00: Thread 11: finishing

Started 8 at 1620651351.2621378899

Finished 8 at 1620651361.2658278942

17:26:01: Thread 8: finishing

Started 6 at 1620651352.4786870480

Finished 6 at 1620651362.4896230698

17:26:02: Thread 6: finishing

17:26:02: Main    : thread 6 done

17:26:02: Main    : before joining thread 7.

17:26:02: Main    : thread 7 done

17:26:02: Main    : before joining thread 8.

17:26:02: Main    : thread 8 done

17:26:02: Main    : before joining thread 9.

17:26:02: Main    : thread 9 done

17:26:02: Main    : before joining thread 10.

17:26:02: Main    : thread 10 done

17:26:02: Main    : before joining thread 11.

17:26:02: Main    : thread 11 done

17:26:02: Main    : before joining thread 12.

17:26:02: Main    : thread 12 done

17:26:02: Main    : before joining thread 13.

17:26:02: Main    : thread 13 done

17:26:02: Main    : before joining thread 14.

17:26:02: Main    : thread 14 done

17:26:02: Main    : before joining thread 15.

17:26:02: Main    : thread 15 done

17:26:02: Main    : before joining thread 16.

17:26:02: Main    : thread 16 done

17:26:02: Main    : before joining thread 17.

17:26:02: Main    : thread 17 done

17:26:02: Main    : before joining thread 18.

17:26:02: Main    : thread 18 done

17:26:02: Main    : before joining thread 19.

17:26:02: Main    : thread 19 done

17:26:02: Main    : before joining thread 20.

17:26:02: Main    : thread 20 done

17:26:02: Main    : before joining thread 21.

17:26:02: Main    : thread 21 done

17:26:02: Main    : before joining thread 22.

17:26:02: Main    : thread 22 done

17:26:02: Main    : before joining thread 23.

17:26:02: Main    : thread 23 done

17:26:02: Main    : before joining thread 24.

17:26:02: Main    : thread 24 done

17:26:02: Main    : before joining thread 25.

17:26:02: Main    : thread 25 done

17:26:02: Main    : before joining thread 26.

17:26:02: Main    : thread 26 done

17:26:02: Main    : before joining thread 27.

17:26:02: Main    : thread 27 done

17:26:02: Main    : before joining thread 28.

17:26:02: Main    : thread 28 done

17:26:02: Main    : before joining thread 29.

17:26:02: Main    : thread 29 done

17:26:02: Main    : before joining thread 30.

17:26:02: Main    : thread 30 done

17:26:02: Main    : before joining thread 31.

17:26:02: Main    : thread 31 done

Took 36.33210492134094

#+end_example

** Running in the Background

The =z_background= function allows you to execute shell commands asynchronously in a new thread. It starts a new Zsh instance and runs the given command without blocking your main Python thread. This is particularly useful when you want to perform non-blocking operations or execute long-running shell commands without interrupting your Python program's flow.

#+begin_src jupyter-python :kernel py_base :session emacs_py_1 :async yes :exports both

from brish import z_background

msg = "You can’t make an omelet without breaking a few eggs."

result_future = z_background(

    "say {msg}",

    # Needs the `say` command, available by default on macOS

)

result_future

#+end_src

#+RESULTS:

: 

The =z_background= function returns a =Future= object.

#+begin_src jupyter-python :kernel py_base :session emacs_py_1 :async yes :exports both

result_future.result()

#+end_src

#+RESULTS:

: CmdResult(retcode=0, out='', err='', cmd=" say 'You can’t make an omelet without breaking a few eggs.' ", cmd_stdin='')

Here is another example:

#+begin_src jupyter-python :kernel py_base :session emacs_py_1 :async yes :exports both

from brish import z_background

import concurrent.futures

# Define multiple commands

commands = [

    "sleep 5 && echo 'First command completed.'",

    "sleep 3 && echo 'Second command completed.'",

    "sleep 1 && echo 'Third command completed.'",

]

# Execute all commands asynchronously

futures = [z_background(cmd) for cmd in commands]

for future in concurrent.futures.as_completed(futures):

    result = future.result()

    print(result.out)

#+end_src

#+RESULTS:

: Third command completed.

: 

: Second command completed.

: 

: First command completed.

: 

* Security Considerations

I am not a security expert, and security doesn't come by default in these situations. So be careful if you use untrusted input in the commands fed to zsh. Nevertheless, I can't imagine any (non-obvious) attack vectors, as the input gets automatically escaped by default. Feedback by security experts will be appreciated.

Note that you can create security holes for yourself, by, e.g., running =eval= on user input:

#+begin_src python :session p1 :results value :exports both

untrusted_input = " ; echo do evil | cat"

z("eval {untrusted_input}") # unsafe

#+end_src

#+RESULTS:

#+begin_example

do evil

#+end_example

#+begin_src python :session p1 :results value :exports both

z("echo {untrusted_input}") # safe

#+end_src

#+RESULTS:

#+begin_example

 ; echo do evil | cat

#+end_example

# One thing to keep in mind is that Brish purposely uses the zsh from your PATH. That zsh will load its dotfiles as usual.

* Known Issues

- Piping binary (non-text) output from zsh to Python does not work

- Nonstandard encodings (non UTF-8) are corrupted

  #+begin_src python :session p1 :results value :exports both

  z("echo 'sth × another (ver.-)'")

  #+end_src

  #+RESULTS:

  #+begin_example

  sth Ã\xb7 another (ver.-)

  #+end_example

- There is always sth piped to the standard input (an empty string by default). This can alter the behavior of some commands such as =ripgrep=; Using =