Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

https://github.com/bendudson/py4cl

Call python from Common Lisp
https://github.com/bendudson/py4cl

common-lisp python

Last synced: 2 months ago
JSON representation

Call python from Common Lisp

Lists

README 

        
[[https://travis-ci.org/bendudson/py4cl][https://travis-ci.org/bendudson/py4cl.svg?branch=master]]



* Introduction



Py4CL is a bridge between Common Lisp and Python, which enables Common

Lisp to interact with Python code. It uses streams to communicate with

a separate python process, the approach taken by [[https://github.com/marcoheisig/cl4py][cl4py]]. This is

different to the CFFI approach used by [[https://github.com/pinterface/burgled-batteries][burgled-batteries]], but has the

same goal. 



** Installing



Depends on:



 - Currently tested with SBCL, CCL and ECL (after 2016-09-06). CLISP

   doesn't (yet) have =uiop:launch-program=.

 - ASDF3 version 3.2.0 (Jan 2017) or later, as =uiop:launch-program=

   is used to run and communicate with python asyncronously.

 - [[https://common-lisp.net/project/trivial-garbage/][Trivial-garbage]], available through Quicklisp.

 - Python 2 or 3

 - (optional) The [[http://www.numpy.org/][NumPy]] python library for multidimensional arrays



Clone this repository into =~/quicklisp/local-projects/= or other

location where it can be found by ASDF:

#+BEGIN_SRC bash

$ git clone https://github.com/bendudson/py4cl.git

#+END_SRC



then load into Lisp with

#+BEGIN_SRC lisp

(ql:quickload :py4cl)

#+END_SRC



** Tests



Tests use [[https://github.com/tgutu/clunit][clunit]], and run on [[https://travis-ci.org/][Travis]] using [[https://github.com/luismbo/cl-travis][cl-travis]]. Most development

is done under Arch linux with SBCL and Python3. To run the tests

yourself:

#+BEGIN_SRC lisp

(asdf:test-system :py4cl)

#+END_SRC

or

#+BEGIN_SRC lisp

(ql:quickload :py4cl/tests)

(py4cl/tests:run)

#+END_SRC



* Examples



Py4CL allows python modules to be imported as Lisp packages, python

functions to be called from lisp, and lisp functions called from

python. In the example below, [[https://www.scipy.org/][SciPy]]'s [[https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.odeint.html][odeint]] function is used to

integrate ODEs defined by a Lisp function. The result is a Lisp array,

which is then plotted using the [[https://matplotlib.org/][matplotlib]] plotting library.



#+BEGIN_SRC lisp

(ql:quickload :py4cl)



(py4cl:import-module "numpy" :as "np")

(py4cl:import-module "scipy.integrate" :as "integrate")



;; Integrate some ODEs

(defparameter *data*

  (integrate:odeint 

   (lambda (y time) 

     (vector (aref y 1)       ; dy[0]/dt = y[1]

             (- (aref y 0)))) ; dy[1]/dt = -y[0]

   #(1.0 0.0)   ; Initial state

   (np:linspace 0.0 (* 2 pi) 20)))  ; Vector of times



; (array-dimensions *data*) => (20 2)



;; Make a plot, save and show it in a window

(py4cl:import-module "matplotlib.pyplot" :as "plt")



(plt:plot *data*)

(plt:xlabel "Time")

(plt:savefig "result.pdf")

(plt:show)

#+END_SRC



More detailed examples of using python packages using =py4cl=:

 - [[./docs/numpy.org][Numpy arrays]]

 - [[./docs/matplotlib.org][Matplotlib plotting]]

 - [[./docs/scipy.org][Scipy scientific library]]



* Reference

** Direct evaluation of python code: =python-eval=, =python-exec=



For direct access to the python subprocess, =python-eval=

evaluates an expression, converting the result to a suitable lisp

type. Note that there are nicer, more lispy wrappers around this function,

described below, but they are mostly built on top of =python-eval=.



#+BEGIN_SRC lisp

(asdf:load-system "py4cl")



(py4cl:python-eval "[i**2 for i in range(5)]") ; => #(0 1 4 9 16)

#+END_SRC



#+RESULTS:

| 0 | 1 | 4 | 9 | 16 |



#+BEGIN_SRC lisp

(py4cl:python-eval "{'hello':'world', 'answer':42}") ; => #

#+END_SRC



#+RESULTS:

: #



Data is passed between python and lisp as text. The python function

=lispify= converts values to a form which can be read by the lisp

reader; the lisp function =pythonize= outputs strings which can be

=eval='d in python. The following type conversions are done:



| Lisp type | Python type        |

|-----------+--------------------|

| NIL       | None               |

| integer   | int                |

| ratio     | fractions.Fraction |

| real      | float              |

| complex   | complex float      |

| string    | str                |

| hash map  | dict               |

| list      | tuple              |

| vector    | list               |

| array     | NumPy array        |

| symbol    | Symbol class       |

| function  | function           |



Note that python does not have all the numerical types which lisp has,

for example complex integers.



Because =python-eval= and =python-exec= evaluate strings as python

expressions, strings passed to them are not escaped or converted as

other types are. To pass a string to python as an argument, call =py4cl::pythonize=



#+BEGIN_SRC lisp

(let ((my-str "testing"))

  (py4cl:python-eval "len(" (py4cl::pythonize my-str) ")" ))

#+END_SRC



#+RESULTS:

: 7



Note that this escaping is done automatically by higher-level interfaces like

=python-call= and =chain=:

#+BEGIN_SRC lisp

(let ((my-str "testing"))

  (py4cl:python-call "len" my-str))

#+END_SRC



#+RESULTS:

: 7



#+BEGIN_SRC lisp

(let ((my-str "testing"))

  (py4cl:chain (len my-str)))

#+END_SRC



#+RESULTS:

: 7



If python objects cannot be converted into a lisp value, then they are

stored and a handle is returned to lisp. This handle can be used to

manipulate the object, and when it is garbage collected the python

object is also deleted (using the [[https://common-lisp.net/project/trivial-garbage/][trivial-garbage]] package).



#+BEGIN_SRC lisp

(destructuring-bind (fig ax) (plt:subplots)

  ;; fig is #S(PY4CL::PYTHON-OBJECT :TYPE "" :HANDLE 6)

  (py4cl:python-eval ax ".plot(" #(0 1 0 1) ")")

  (plt:show)) 

#+END_SRC



The interface to python objects is nicer using =chain= (see below):

#+BEGIN_SRC lisp

(destructuring-bind (fig ax) (plt:subplots)

  (py4cl:chain ax (plot #(0 1 0 1)))

  (plt:show)) 

#+END_SRC



The python process can be explicitly started and stopped using

=python-start= and =python-stop=, but =py4cl= functions start python

automatically if needed by calling =python-start-if-not-alive=.



** Calling python functions: =python-call=



=python-call= can be used to pass arguments to any python callable, 

such as a function in a module:



#+BEGIN_SRC lisp

(py4cl:python-exec "import math")

(py4cl:python-call "math.sqrt" 42)

#+END_SRC



#+RESULTS:

: 6.4807405



or a lambda function:

#+BEGIN_SRC lisp

(py4cl:python-call "lambda x: 2*x" 21)

#+END_SRC



#+RESULTS:

: 42



Keywords are translated, with the symbol made lowercase:

#+BEGIN_SRC lisp

(py4cl:python-call "lambda a=0, b=1: a-b" :b 2 :a 1)

#+END_SRC



#+RESULTS:

: -1



** Calling python methods: =python-method=



Python methods on objects can be called by using the =python-method= function. The first argument

is the object (including strings, arrays, tuples); the second argument is either a string or a symbol

specifying the method, followed by any arguments:

#+BEGIN_SRC lisp

(py4cl:python-method "hello {0}" 'format "world") ; => "hello world"

#+END_SRC



#+RESULTS:

: hello world



#+BEGIN_SRC lisp

(py4cl:python-method '(1 2 3) '__len__) ; => 3

#+END_SRC



#+RESULTS:

: 3



** Getting python attributes: =python-getattr=



The attributes of a python object can be accessed using the generic

functon =python-getattr=, with the python object as first argument and

a string as the name of the attribute:

#+BEGIN_SRC lisp

(py4cl:python-getattr '(1 2 3) "__doc__")

#+END_SRC



Note: Methods for this function can also be defined for lisp classes,

enabling python code to access attributes of lisp objects. See below

for details.



** Chaining python methods: =chain=



In python it is quite common to apply a chain of method calls, data

member access, and indexing operations to an object. To make this work

smoothly in Lisp, there is the =chain= macro (Thanks to @kat-co and

[[https://common-lisp.net/project/parenscript/reference.html][parenscript]] for the inspiration). This consists of a target object,

followed by a chain of operations to apply.  For example

#+BEGIN_SRC lisp

(py4cl:chain "hello {0}" (format "world") (capitalize)) ; => "Hello world"

#+END_SRC



#+RESULTS:

: Hello world



which is converted to python 

#+BEGIN_SRC python

return "hello {0}".format("world").capitalize()

#+END_SRC



#+RESULTS:

: Hello world



The only things which are treated specially by this macro are lists

and symbols at the top level. The first element of lists are treated as

python method names, top-level symbols are treated as data

members. Everything else is evaluated as lisp before being converted

to a python value.



If the first argument is a list, then it is assumed to be a python

function to be called; otherwise it is evaluated before converting to

a python value. For example

#+BEGIN_SRC lisp

(py4cl:chain (slice 3) stop)

#+END_SRC



#+RESULTS:

: 3



is converted to the python:

#+BEGIN_SRC python

return slice(3).stop

#+END_SRC



#+RESULTS:

: 3



Symbols as first argument, or arguments to python methods, are

evaluated, so the following works:

#+BEGIN_SRC lisp

(let ((format-str "hello {0}")

      (argument "world"))

 (py4cl:chain format-str (format argument))) ; => "hello world"

#+END_SRC



#+RESULTS:

: hello world



Arguments to methods are lisp, since only the top level forms in =chain= are treated specially:

#+BEGIN_SRC lisp

(py4cl:chain "result: {0}" (format (+ 1 2))) ; => "result: 3"

#+END_SRC



#+RESULTS:

: result: 3



Indexing with =[]= brackets is commonly used in python, which calls the =__getitem__= method.

This method can be called like any other method

#+BEGIN_SRC lisp

(py4cl:chain "hello" (__getitem__ 4)) ; => "o"

#+END_SRC



#+RESULTS:

: o



but since this is a common method an alias =[]= is supported:

#+BEGIN_SRC lisp

(py4cl:chain "hello" ([] 4)) ; => "o"

#+END_SRC



#+RESULTS:

: o



which is converted to the python

#+BEGIN_SRC python

return "hello"[4]

#+END_SRC



#+RESULTS:

: o



For simple cases where the index is a value like a number or string

(not a symbol or a list), the brackets can be omitted:

#+BEGIN_SRC lisp

(py4cl:chain "hello" 4) ; => "o"

#+END_SRC



#+RESULTS:

: o



Slicing can be done by calling the python =slice= function:

#+BEGIN_SRC lisp

(py4cl:chain "hello" ([] (py4cl:python-call "slice" 2 4)))  ; => "ll"

#+END_SRC



#+RESULTS:

: ll



which could be imported as a lisp function (see below):

#+BEGIN_SRC lisp

(py4cl:import-function "slice")

(py4cl:chain "hello" ([] (slice 2 4))) ; => "ll"

#+END_SRC



#+RESULTS:

: ll



This of course also works with multidimensional arrays:

#+BEGIN_SRC lisp

(py4cl:chain #2A((1 2 3) (4 5 6))  ([] 1 (slice 0 2)))  ;=> #(4 5)

#+END_SRC



#+RESULTS:

| 4 | 5 |



Sometimes the python functions or methods may contain upper case

characters; class names often start with a capital letter. All symbols

are converted to lower case, but the case can be controlled by passing

a string rather than a symbol as the first element:

#+BEGIN_SRC lisp

;; Define a class

(py4cl:python-exec

   "class TestClass:

      def doThing(self, value = 42):

        return value")



;; Create an object and call the method

(py4cl:chain ("TestClass") ("doThing" :value 31))  ; => 31

#+END_SRC

Note that the keyword is converted, converting to lower case.



** Printing from python



Since standard output is used for communication between lisp and python, this is

redirected (to a =StringIO= buffer) while user python code is running. The

output from python functions is then sent to lisp, to be printed to

=*standard-output*=.  This means that anything printed by the python process may

only appear in chunks, as it is sent to lisp. The following does however work as

expected:

#+BEGIN_SRC lisp :results output

(py4cl:chain (print "hello world")) 

; => prints "hello world", returns NIL

#+END_SRC



#+RESULTS:

: hello world



In python =print_function= is imported from =__future__=, so should be available

as a function in python 2.6+, as well as in version 3+.



** Asynchronous python functions: =python-call-async=



One of the advantages of using streams to communicate with a separate

python process, is that the python and lisp processes can run at the

same time. =python-call-async= calls python but returns a closure

immediately. The python process continues running, and the result can

be retrieved by calling the returned closure. 



#+BEGIN_SRC lisp

(defparameter thunk (py4cl:python-call-async "lambda x: 2*x" 21))



(funcall thunk)  ; => 42

#+END_SRC



#+RESULTS:

: 42



If the function call requires callbacks to lisp, then these will only

be serviced when a =py4cl= function is called. In that case the python

function may not be able to finish until the thunk is called. This

should not result in deadlocks, because all =py4cl= functions can

service callbacks while waiting for a result.



** Importing functions: =import-function=



Python functions can be made available in Lisp by using =import-function=. By

default this makes a function which can take any number of arguments, and then

translates these into a call to the python function.

#+BEGIN_SRC lisp

(asdf:load-system "py4cl")



(py4cl:python-exec "import math")

(py4cl:import-function "math.sqrt")

(math.sqrt 42) ; => 6.4807405

#+END_SRC



#+RESULTS:

: 6.4807405



If a different symbol is needed in Lisp then the =:as= keyword can be

used with either a string or symbol:

#+BEGIN_SRC lisp

(py4cl:import-function "sum" :as "pysum")

(pysum '(1 2 3))  ; => 6

#+END_SRC



#+RESULTS:

: 6



This is implemented as a macro which defines a function which in turn calls =python-call=.



** Importing modules: =import-module=



Python modules can be imported as lisp packages using =import-module=.

For example, to import the [[https://matplotlib.org/][matplotlib]] plotting library, and make its functions

available in the package =PLT= from within Lisp:

#+BEGIN_SRC lisp :session import-example

(asdf:load-system "py4cl")

(py4cl:import-module "matplotlib.pyplot" :as "plt") ; Creates PLT package

#+END_SRC



#+RESULTS:

: T



This will also import it into the python process as the module =plt=, so that

=python-call= or =python-eval= can also make use of the =plt= module. 



Like =python-exec=, =python-call= and other similar functions, 

=import-module= starts python if it is not already running, so that

the available functions can be discovered.



The python docstrings are made available as Lisp function docstrings, so we can see them

using =describe=:

#+BEGIN_SRC  lisp :session import-example

(describe 'plt:plot)

#+END_SRC



Functions in the =PLT= package can be used to make simple plots:

#+BEGIN_SRC lisp :session import-example

(plt:plot #(1 2 3 2 1) :color "r")

(plt:show)

#+END_SRC



#+RESULTS:

: NIL



Notes:

 -  =import-module= should be used as a top-level form, to ensure that

   the package is defined before it is used.



- If using =import-module= within [[https://orgmode.org/worg/org-contrib/babel/][org-mode babel]] then the import

  should be done in a separate code block to the first use of the

  imported package, or a condition will be raised like "Package NP

  does not exist."



** Exporting a function to python: =export-function=



Lisp functions can be passed as arguments to =python-call= 

or imported functions:

#+BEGIN_SRC lisp

(py4cl:python-exec "from scipy.integrate import romberg")



(py4cl:python-call "romberg" 

                   (lambda (x) (/ (exp (- (* x x)))

                                  (sqrt pi)))

                   0.0 1.0) ; Range of integration

#+END_SRC



#+RESULTS:

: 0.4213504



Lisp functions can be made available to python code using =export-function=:

#+BEGIN_SRC lisp

(py4cl:python-exec "from scipy.integrate import romberg")



(py4cl:export-function (lambda (x) (/ (exp (- (* x x)))

                                      (sqrt pi))) "gaussian")



(py4cl:python-eval "romberg(gaussian, 0.0, 1.0)") ; => 0.4213504

#+END_SRC



#+RESULTS:

: 0.4213504



** Manipulating objects remotely: =remote-objects=



If a sequence of python functions and methods are being used to manipulate data,

then data may be passed between python and lisp. This is fine for small amounts

of data, but inefficient for large datasets.



The =remote-objects= and =remote-objects*= macros provide =unwind-protect= environments

in which all python functions return handles rather than values to lisp. This enables

python functions to be combined without transferring much data.



The difference between these macros is =remote-objects= returns a handle, but

=remote-objects*= evaluates the result, and so will return a value if possible.



#+BEGIN_SRC lisp

(py4cl:remote-objects (py4cl:python-eval "1+2")) ; => #S(PY4CL::PYTHON-OBJECT :TYPE "" :HANDLE 0)

#+END_SRC



#+RESULTS:

: #S(PY4CL::PYTHON-OBJECT :TYPE "" :HANDLE 4)



#+BEGIN_SRC lisp

(py4cl:remote-objects* (py4cl:python-eval "1+2")) ; => 3

#+END_SRC



#+RESULTS:

: 3



The advantage comes when dealing with large arrays or other datasets:

#+BEGIN_SRC lisp

(time (np:sum (np:arange 1000000)))

; => 3.672 seconds of real time

;    390,958,896 bytes consed

#+END_SRC



#+BEGIN_SRC lisp

(time (py4cl:remote-objects* (np:sum (np:arange 1000000))))

; => 0.025 seconds of real time

;    32,544 bytes consed

#+END_SRC

** =setf=-able places



The =python-eval= function is =setf=-able, so that python objects can

be assigned to by using =setf=. Since =chain= uses =python-eval=, it is also

=setf=-able. This can be used to set elements in an array, entries in a dict/hash-table, 

or object data members, for example:

#+BEGIN_SRC lisp

(py4cl:import-module "numpy" :as "np")

#+END_SRC



#+RESULTS:

: T



#+BEGIN_SRC lisp

(py4cl:remote-objects*

  (let ((array (np:zeros '(2 2))))

    (setf (py4cl:chain array ([] 0 1)) 1.0

          (py4cl:chain array ([] 1 0)) -1.0)

    array)) 

; => #2A((0.0 1.0)

;        (-1.0 0.0))

#+END_SRC



#+RESULTS:

: #2A((0.0 1.0) (-1.0 0.0))



Note that this modifies the value in python, so the above example only

works because =array= is a handle to a python object, rather than an

array which is stored in lisp. The following therefore does not work:

#+BEGIN_SRC lisp

(let ((array (np:zeros '(2 2))))

  (setf (py4cl:chain array ([] 0 1)) 1.0

        (py4cl:chain array ([] 1 0)) -1.0)

  array)

; => #2A((0.0 0.0)

;        (0.0 0.0))

#+END_SRC



#+RESULTS:

: #2A((0.0 0.0) (0.0 0.0))



The =np:zeros= function returned an array to lisp; the array was then

sent to python and modified in python. The modified array is not

returned, since this would mean transferring the whole array. If the

value is in lisp then just use the lisp functions:

#+BEGIN_SRC lisp

(let ((array (np:zeros '(2 2))))

  (setf (aref array 0 1) 1.0

        (aref array 1 0) -1.0)

  array)

; => #2A((0.0 1.0)

;        (-1.0 0.0))

#+END_SRC



#+RESULTS:

: #2A((0.0 1.0) (-1.0 0.0))



** Passing lisp objects to python: =python-getattr=



Lisp structs and class objects can be passed to python, put into data structures and

returned:

#+BEGIN_SRC lisp

(py4cl:import-function "dict") ; Makes python dictionaries



(defstruct test-struct 

    x y)



(let ((map (dict :key (make-test-struct :x 1 :y 2))))  ; Make a dictionary, return as hash-map

  ;; Get the struct from the hash-map, and get the Y slot

  (test-struct-y

    (py4cl:chain map "key")))  ; => 2

#+END_SRC



#+RESULTS:

: 2



In python this is handled using an object of class =UnknownLispObject=, which

contains a handle. The lisp object is stored in a hash map

=*lisp-objects*=. When the python object is deleted, a message is sent to remove

the object from the hash map.



To enable python to access slots, or call methods on a struct or class, a

handler function needs to be registered. This is done by providing a method 

for generic function =python-getattr=. This function will be called when a

python function attempts to access attributes of an object (=__getattr__=

method).



#+BEGIN_SRC lisp

;; Define a class with some slots

(defclass test-class ()

  ((value :initarg :value)))



;; Define a method to handle calls from python

(defmethod py4cl:python-getattr ((object test-class) slot-name)

  (cond

    ((string= slot-name "value") ; data member

      (slot-value object 'value))

    ((string= slot-name "func")  ; method, return a function

      (lambda (arg) (* 2 arg)))

    (t (call-next-method)))) ; Otherwise go to next method



(let ((instance (make-instance 'test-class :value 21))) 

  ;; Get the value from the slot, call the method

  ;; python: instance.func(instance.value)

  (py4cl:chain instance (func (py4cl:chain instance value))))  ; => 42

#+END_SRC



#+RESULTS:

: 42



Inheritance then works as usual with CLOS methods:

#+BEGIN_SRC lisp

;; Class inheriting from test-class

(defclass child-class (test-class)

  ((other :initarg :other)))



;; Define method which passes to the next method if slot not recognised

(defmethod py4cl:python-getattr ((object child-class) slot-name)

  (cond

    ((string= slot-name "other")

     (slot-value object 'other))

    (t (call-next-method))))



(let ((object (make-instance 'child-class :value 42 :other 3)))

  (list 

    (py4cl:chain object value) ; Call TEST-CLASS getattr method via CALL-NEXT-METHOD

    (py4cl:chain object other))) ;=> (42 3)

#+END_SRC



#+RESULTS:

| 42 | 3 |