Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/aartaka/graven-image

Portability library for better interaction and debugging of a running Common Lisp image through text REPL.
https://github.com/aartaka/graven-image

benchmarking common-lisp debugging garbage-collection help inspection lisp memory reflection

Last synced: about 1 month ago
JSON representation

Portability library for better interaction and debugging of a running Common Lisp image through text REPL.

Awesome Lists containing this project

README

        

#+TITLE:Graven Image

#+begin_quote
Thou shalt not make unto thee any graven image, or any likeness of any
thing that is in heaven above, or that is in the earth beneath, or
that is in the water under the earth.
#+end_quote

Graven Image is a Common Lisp portability library (a less fancier name
might've been =trivial-debugging=) for better interaction and
debugging of a running Lisp image. One can inspect and debug all the
things under heaven and [[https://www.corecursive.com/lisp-in-space-with-ron-garret/][above it]]—all in their own REPL-resident Lisp
image!

Graven Image reuses compiler internals to improve/redefine the
existing standard functions. This "improvement" often comes at a cost
of changing the API of a function (for better customizability) or
making it slightly less reliable (due to unstable, compiler-internal,
or otherwise hacky implementation.)

NOTE: Graven Image is currently being refactored into more focused libraries.
Like [[https://github.com/aartaka/trivial-time][trivial-time]] and [[https://github.com/aartaka/trivial-inspect][trivial-inspect]].

The library is purposefully limited in scope:
- Improving the standard functions has a priority over introducing new
ones.
- [[https://github.com/m-n/repl-utilities][repl-utilities]] as a contrasting approach: it defines lots of
"DWIM" functions, significantly altering the REPL interaction.
- There are helpers like =function-lambda-list*= in Graven Image,
but these are merely aliases/one-liners over standard/improved
functions.
- I get carried away sometimes, though. =benchmark*= is one of
such [[https://xkcd.com/356/][nerd snipes]]. But still, it's quite close to the underlying
=with-time*= 😉
- The interaction paradigm of the improved functions should stay
standard (i.e. use =*query-io*= where standard requires
=*query-io*=.)
- This is to ensure that libraries like [[https://github.com/atlas-engineer/ndebug/][Ndebug]] can rely on standard
facilities safely when used with Graven Image.
- Function arglist can be modified for convenience, but it should
preferably stay as close to standard/implementation-specific arglist
as possible.
- Graven Image strives to not modify the REPL/image/shell in any way,
relying on implementation defaults instead.
- [[https://github.com/ciel-lang/CIEL][CIEL]] is taking a different direction: redefining the REPL for
increased utility.
- [[https://github.com/vseloved/flight-recorder][flight-recorder]] uses a separate shell script to add a new
functionality.
- Portably reusing implementation-specific functionality is better
than re-implementing it. But if some functionality is e.g. unique to
SBCL, it might be dropped as non-portable.
- [[https://github.com/TeMPOraL/tracer][tracer]], [[https://github.com/fukamachi/supertrace][supertrace]], [[https://github.com/40ants/cl-flamegraph][cl-framegraph]], and other non-portable
SBCL-specific improvements.

* Getting started

Clone the Git repository:
#+begin_src sh
git clone --recursive https://github.com/aartaka/graven-image ~/common-lisp/
#+end_src

And load =:graven-image= in the REPL:
#+begin_src lisp
(asdf:load-system :graven-image)
;; or, if you use Quicklisp
(ql:quickload :graven-image)
#+end_src

You can also install Graven Image via Guix, using the bundled
=guix.scm= file:
#+begin_src sh
guix package -f guix.scm
#+end_src

Someday (after at least minor version bump) I'll send a patch to Guix
so that you can also install it with =guix install=. Until then—stay
tuned (or send the patch yourself—I don't mind.)

* Convenient =use=

If you want your REPL to start with Graven Image constructs accessible
without package prefix, then simply use the package directly.
#+begin_src lisp
(asdf:load-system :graven-image)
;; Imports external symbols of Graven Image into the current package
;; (CL-USER?) Safe, because Graven Image shadows no CL symbols.
(use-package :graven-image)
#+end_src

Or, if you're more orderly and disciplined about packages than I am,
you can always use =trivial-package-local-nicknames= and define a
shorter Graven Image nickname:
#+begin_src lisp
(trivial-package-local-nicknames:add-package-local-nickname
:g :graven-image :cl-user)
#+end_src

If you want to replace the standard utilities with Graven Image ones,
you can safely do so with the familiar:
#+begin_src lisp
;; For functions
(fmakunbound 'apropos)
(setf (fdefinition 'apropos) (fdefinition 'gimage:apropos*))
;; For macros
(setf (macro-function 'time) (macro-function 'gimage:time*))
#+end_src

* Enhanced functions (mostly from ANSI CL [[https://cl-community-spec.github.io/pages/Debugging-Utilities.html][Debugging Utilities]] chapter)

The functions that Graven Image exposes are safely =:use=-able
star-appended functions/macros (i.e. =describe*= instead of
=describe=). Currently improved ones are:
- =y-or-n-p*= / =yes-or-no-p*=
- =apropos*= / =apropos-list*=
- =function-lambda-expression*=
- =time*=
- =describe*= / =inspect*=
- =dribble*=
- =documentation*=

All of the functions exposed by Graven Image are generics, so one can
easily define =:around= and other qualified methods for them.

** =y-or-n-p*=, =yes-or-no-p*= (generic functions)

Signature:
#+begin_src lisp
y-or-n-p* &optional control &rest arguments => generalized-boolean
yes-or-no-p* &optional control &rest arguments => generalized-boolean
#+end_src

Improvements are:
- Both functions accept options from =graven-image:*yes-or-no-options*=, thus
allowing for "nope" or "ay" to be valid responses too.
- Both functions mean the same now, because it makes no sense in
differentiating them (and because most Emacs users use a magical
=(fset 'yes-or-no-p 'y-or-n-p)= in their config, setting the
precedent for shorter yes/no queries).
- No beeps (just define a =yes-or-no-p* :before= method to add beeps
if you like 'em; see the "Customization" section below).

** =apropos-list*=, =apropos*= (generic functions)

Signature:
#+begin_src lisp
apropos-list* string &optional (package nil) exported-only docs-too => list of symbols
apropos* string &optional (package nil) exported-only docs-too => no values
#+end_src

=apropos-list*= now allows listing exported symbols only (with
=exported-only=), which was a non-portable privilege of SBCL/Allegro
until now. Search over docs (more intuitive for =apropos(-list)*= than
mere name search) is possible with =docs-too=.

Based on this foundation, =apropos*= lists symbols with their types,
values, and documentation, so that implementation-specific formats are
gone for a better and more unified listing:

#+begin_src lisp
(apropos* :max)
;; MAX [FUNCTION (NUMBER &REST
;; MORE-NUMBERS) : Return the greatest of its arguments; among EQUALP greatest, return...]
;; :MAX [SELF-EVALUATING]
;; CFFI::MAX-ALIGN
;; SB-ASSEM::MAX-ALIGNMENT [CONSTANT = 5]
;; ...
;; SB-C::MAXES
;; ALEXANDRIA:MAXF [MACRO (#:PLACE &REST NUMBERS) : Modify-macro for MAX. Sets place designated by the first argument to the...]
;; SB-KERNEL::MAXIMAL-BITMAP
;; ...
;; SB-LOOP::LOOP-ACCUMULATE-MINIMAX-VALUE [MACRO (LM OPERATION FORM)]
;; SB-LOOP::LOOP-MAXMIN-COLLECTION [FUNCTION (SPECIFICALLY)]
;; SB-LOOP::LOOP-MINIMAX [CLASS (STRUCTURE-OBJECT)]
;; ...
#+end_src

** =function-lambda-expression*= (generic function)

Signature:
#+begin_src lisp
function-lambda-expression* function/macro/method/symbol &optional force => list, list, symbol, list
;; Alias:
lambda-expression* function/macro/method/symbol &optional force => list, list, symbol, list
#+end_src

This function tries to read source files, process the definitions of
functions, and build at least a barebones lambda from the arglist and
documentation of the function. So that CL =function-lambda-expression=
returns:
#+begin_src lisp
(function-lambda-expression #'identity)
;; => NIL, T, IDENTITY
(function-lambda-expression #'print-object)
;; => NIL, T, PRINT-OBJECT
#+end_src

While the new Graven Image =function-lambda-expression= now returns:
#+begin_src lisp
(function-lambda-expression* #'idenitity)
;; => (LAMBDA (THING) "This function simply returns what was passed to it." THING),
;; NIL, IDENTITY, (FUNCTION (T) (VALUES T &OPTIONAL))
(function-lambda-expression* #'print-object t) ; Notice the T for FORCE, to build a dummy lambda.
;; => (LAMBDA (SB-PCL::OBJECT STREAM)), NIL, PRINT-OBJECT, (FUNCTION (T T) *)
#+end_src

Which means:
- =identity= is actually not a closure, and has a reliable source!
- =print-object= is a generic and thus is not really inspectable, so
we build a dummy lambda for it when =force= argument is provided.
- This might be a questionable choice, but it at least allows us to
get function arglists from =function-lambda-expression= in a
portable-ish way. The standard doesn't provide us with much ways
to know an arglist of a function beside this.

*** Return values

Things that =function-lambda-expression*= now returns are:
- Lambda expression.
- For lambda functions, their source.
- For regular functions, their =defun= turned into a =lambda=.
- For anything else, a constructed empty =(lambda (arglist...)
documentation nil)= (only when =force= is T).
- Or, in case all the rest fails, NIL.
- Whether the thing is a closure
- If it is, might return an alist of the actual closed-over values,
whenever accessible (not for all implementations).
- If closed-over values are not accessible, returns T.
- If it's not a closure, returns NIL.
- Function name. Mostly falls back to the standard
=function-lambda-expression=, but also inspects
implementation-specific function objects if necessary.
- Function type, whenever accessible (SBCL and ECL).

*** Helpers

Based on these new features of =function-lambda-expression*=, here are
some Graven Image-specific helpers:
- =function-lambda-list*= :: Get the lambda list of a function.
- =function-arglist*= :: Alias.
- =lambda-list*= :: Alias for =function-lambda-list*=.
- =arglist*= :: Alias.
- =function-name*= :: Get the name of a function.
- =function-type*= :: Get its ftype.

#+begin_src lisp
function-lambda-list* function => list
function-arglist* function => list
lambda-list* function => list
arglist* function => list
function-name* function => symbol
function-type* function => list
#+end_src

** =time*= (macro)

Signature:
#+begin_src lisp
time* &rest forms => return-values
#+end_src

The improved =time*= from Graven Image reuses as much
implementation-specific APIs as possible, with the predictable output
format.

And it also allows providing several forms, yay!

*** =benchmark*= (macro)

Signature:
#+begin_src lisp
benchmark* (&optional (repeat 1000)) &body forms => return-values
#+end_src

While =time*= is the standard benchmarking/profiling solution, it's
almost always too simple for proper benchmarking. Most systems getting
complex enough end up with some form of custom
benchmarking. Shinmera's [[https://github.com/Shinmera/trivial-benchmark/][trivial-benchmark]] is one such example. Graven
Image =benchmark*= is heavily inspired by =trivial-benchmark=, but has
a more portable foundation in the form of =with-time*=.

As many other benchmarking macros, =benchmark*= repeats its body a
certain number of times, collecting timing stats for every run, and
then prints aggregate statistics for the total runs.
#+begin_src lisp
(gimage::benchmark* (20) ;; Repeat count.
(loop for i below 1000 collect (make-list i) finally (return 1)))
;; Benchmark for 20 runs of
;; (LOOP FOR I BELOW 1000
;; COLLECT (MAKE-LIST I)
;; FINALLY (RETURN 1))
;; - MINIMUM AVERAGE MAXIMUM TOTAL
;; REAL-TIME 0.0 0.00175 0.019 0.035
;; USER-RUN-TIME 0.000668 0.0016634 0.016315 0.033268
;; SYSTEM-RUN-TIME 0.0 0.00021195 0.003794 0.004239
;; GC-RUN-TIME 0.0 0.00085 0.017 0.017
;; BYTES-ALLOCATED 7997952.0 8008154.5 8030464.0 160163090.0
#+end_src

*** =with-time*= (macro)

Signature:
#+begin_src lisp
with-time* (&rest time-keywords) (&rest multiple-value-args) form &body body
#+end_src

As the implementation detail of =time*= and =benchmark*=, =with-time*=
allows to get the timing data for interactive
querying. =time-keywords= allow =&key=-matching the timing data (like
=:gc= time or bytes =:allocated=) for processing in the body. While
=multiple-value-args= allow matching against the return values of the
=form=. So we get best of the both worlds: timing data and return
values. This flexibility enables =time*=, with its requirements of
printing the data and returning the original values at the same time.

For example, here's how one would track the allocated bytes and
garbage collection times when running a cons-heavy code:
#+begin_src lisp
(gimage:with-time* (&key aborted gc-count gc allocated)
(lists lists-p)
(loop for i below 1000
collect (make-list i :initial-element :hello)
into lists
finally (return (values lists t)))
(unless aborted
(format t "Bytes allocated: ~a, GC ran ~d times for ~a seconds"
allocated gc-count gc)))
;; Bytes allocated: 7997952, GC ran NIL times for 0 seconds
#+end_src

** =describe*= (generic function)

Signature:
#+begin_src lisp
describe* object &optional (stream t) respect-methods
#+end_src

Describes the =object= to the stream, but this time with portable
format of description (determined by =graven-image:description*= and
specified for many standard classes) and with predictable set of
properties (=graven-image:fields*=). In Graven Image, both
=describe= and =inspect= have the same format and the same set of
fields.

As a note of respect to the original =describe=, Graven Image one
allows to reuse the =describe-object= methods defined for user
classes. To enable this, pass T to =respect-methods=.

*** =graven-image:fields*= (generic function)

Signature:
#+begin_src lisp
fields* object &key strip-null &allow-other-keys
#+end_src

Returns an undotted alist of properties for the =object=. Custom
fields provided by Graven Image are named with keywords, while the
implementation-specific ones use whatever the implementation
uses. Arrays and hash-tables are inlined into fields to allow
indexing these right from the inspector.

See =fields*= documentation for more details.

*** =graven-image:description*= (generic function)

Signature:
#+begin_src lisp
description* object &optional stream
#+end_src

Concise and informative description of =object= to the
=stream=. Useful information from most of the implementations
tested—united into one description header.

** =inspect*= (generic function)

Signature:
#+begin_src lisp
inspect* object &optional strip-null
#+end_src

New'n'shiny =inspect*= has:
- Most commands found in other implementation, with familiar names.
- Abbreviations like =H -> HELP= (inspired by SBCL).
- Ability to set object field values with =(:set key value)= command
(inspired by CCL).
- Built-in pagination with ways to scroll it (=:next-page=,
=:previous-page=, =:home=) and change it (=:length=).
- Property indexing by both integer indices and property names (with
abbreviations for them too!).
- Ability to ignore =nil= properties with =strip-null= argument
(inspired by SBCL). On by default!
- And the ability to evaluate arbitrary expressions (with =:evaluate=
command or simply by inputting something that doesn't match any
command).

And here's a help menu of the new =inspect*= (in this case, inspecting
=*readtable*=), just to get you teased:

#+begin_src
This is an interactive interface for 5
Available commands are:
:? Show the instructions for using this interface.
:HELP Show the instructions for using this interface.
:QUIT Exit the interface.
:EXIT Exit the interface.
(:LENGTH NEW) Change the page size.
(:WIDTH NEW) Change the page size.
(:WIDEN NEW) Change the page size.
:NEXT Show the next page of fields (if any).
:PREVIOUS Show the previous page of fields (if any).
:PRINT Print the current page of fields.
:PAGE Print the current page of fields.
:HOME Scroll back to the first page of fields.
:RESET Scroll back to the first page of fields.
:TOP Scroll back to the first page of fields.
:THIS Show the currently inspected object.
:SELF Show the currently inspected object.
:REDISPLAY Show the currently inspected object.
:SHOW Show the currently inspected object.
:CURRENT Show the currently inspected object.
:AGAIN Show the currently inspected object.
(:EVAL EXPRESSION) Evaluate the EXPRESSION.
:UP Go up to the previous level of the interface.
:POP Go up to the previous level of the interface.
:BACK Go up to the previous level of the interface.
(:SET KEY VALUE) Set the KEY-ed field to VALUE.
(:MODIFY KEY VALUE) Set the KEY-ed field to VALUE.
(:ISTEP KEY) Inspect the object under KEY.
(:INSPECT KEY) Inspect the object under KEY.
:STANDARD Print the inspected object readably.
:AESTHETIC Print the inspected object aesthetically.

Possible inputs are:
- Mere symbols: run one of the commands above, matching the symbol.
- If there's no matching command, then match against fields.
- If nothing matches, evaluate the symbol.
- Integer: act on the field indexed by this integer.
- If there are none, evaluate the integer.
- Any other atom: find the field with this atom as a key.
- Evaluate it otherwise.
- S-expression: match the list head against commands and fields,
as above.
- If the list head does not match anything, evaluate the
s-expression.
- Inside this s-expression, you can use the `$' function to fetch
the list of values under provided keys.
#+end_src

** =dribble*= (generic function)

Signature:
#+begin_src lisp
dribble* &optional pathname (if-exists :append)
#+end_src

Dribble the REPL session to =pathname=. Unlike the
implementation-specific =dribble=, this one formats all of the session
as =load=-able Lisp file fully reproducing the session. So all the
input forms are printed verbatim, and all the outputs are commented
out.

Beware: using any interactive function (like =inspect= etc.) breaks
the dribble REPL. But then, it's unlikely one'd want to record
interactive session into a dribble file.

** =documentation*= (generic function)

Signature:
#+begin_src lisp
documentation* object &optional (doc-type t)
doc* object &optional (doc-type t)
#+end_src

Improved version of =documentation=. Two main improvements are:
=doc-type= is now optional, and =doc*= alias is available for
convenience.

documentation.lisp also defines more =documentation= methods (and
respective =setf= method) to simplify documentation fetching and
setting. In particular, method on =(symbol (eql t))= to simplify
symbol documentation search; and =(t (eql 'package))= with a new
doc-type for package documentation convenience.

** =break*= (macro)

Signature:
#+begin_src lisp
break* &rest arguments
#+end_src

A more useful wrapper for =break=, listing the function it's called
from and the provided symbol values. See examples in the docstring.

* Customization

Graven Image is made to be extensible. That's why most of the improved
functions are generic: one can define special methods for their data
and patch the behavior with =:before=, =:after=, and =:around=
methods. Most of Graven Image functions mention the variables/things
influencing them in the docstring. Here's a set of useful
customizations:

** Beeping before =yes-or-no-p*=

Restoring the standard-ish (beeping with bell (ASCII 7) character) behavior:
#+begin_src lisp
(defmethod gimage:yes-or-no-p* :before (&optional control &rest arguments)
(declare (ignore control arguments))
(write-char (code-char 7) *query-io*)
(finish-output *query-io*))
#+end_src

** Changing the accepted yes/no options for =yes-or-no-p*= and =y-or-n-p*=
#+begin_src lisp
;; Make it strict yes/no as per standard.
(defmethod gimage:yes-or-no-p* :around (&optional control &rest arguments)
(declare (ignore control arguments))
(let ((gimage:*yes-or-no-options*
'(("yes" . t)
("no" . nil))))
(call-next-method)))

;; Add more yes/no options (Russian, for example).
(defmethod gimage:y-or-n-p* :around (&optional control &rest arguments)
(declare (ignore control arguments))
(let ((gimage:*yes-or-no-options*
(append
gimage:*yes-or-no-options*
'(("да" . t)
("ага" . t)
("нет" . nil)
("не" . nil)
("неа" . nil)))))
(call-next-method)))
#+end_src

** Sorting =apropos-list*= lists

Implementations are not good at sorting things, and their results are
not often useful. Sorting things the way one needs is a useful
extension. Here's a simple yet effective =:around= method that sorts
things by =string= occurence:
#+begin_src lisp
(defmethod gimage:apropos-list* :around (string &optional packages external-only docs-too)
"Sort symbols by the relation of subSTRING count to the length of symbol."
(declare (ignorable packages external-only docs-too))
(let ((result (call-next-method)))
(sort
(remove-duplicates result)
;; For more comprehensive matching, see
;; a1b4ebd649e0268b1566e80709e7cea41363d006 and other commits
;; before c090d6dc14e05c561cf5c39cf5f6cc02e8cd04c5.
#'> :key (lambda (sym)
(let ((match-count 0))
(uiop:frob-substrings
(string sym) (list (string string))
(lambda (sub frob)
(incf match-count)
(funcall frob sub)))
(/ match-count (length (string sym))))))))
#+end_src

** Changing printer settings for Graven Image output

Graven Image =inspect*= function uses =*interface-lines*= for the
number of properties to list. If your screen is more than 20 lines
high, you might want to add more lines:

#+begin_src lisp
(defmethod gimage:inspect* :around (object)
(declare (ignore object))
(let ((gimage:*interface-lines* 45))
(call-next-method)))
#+end_src

Most of Graven Image functions also rely on
implementation/REPL-specific printer variables, which might be
un-intuitive, overly verbose, or too short. Binding printer variables
around Graven Image functions helps that too:

#+begin_src lisp
(defmethod gimage:apropos* :around (string &optional package external-only docs-too)
(declare (ignore string package external-only docs-too))
;; Note that you can also use
;; `sb-ext:*compiler-print-variable-alist*' and
;; `sb-ext:*debug-print-variable-alist*' on SBCL.
(let ((*print-case* :downcase)
(*print-level* 2)
(*print-lines* 2)
(*print-length* 10))
(call-next-method)))
#+end_src

A noisy apropos function listing like
#+begin_src lisp
X86::*X86-OPERAND-TYPE-NAMES* [VARIABLE = ((:REG8 . 1) (:REG16 . 2) (:REG32 . 4) (:REG64 . 8) (:IMM8 . 16) (:IMM8S . 32) (:IMM16 . 64) (:IMM32 . 128) (:IMM32S . 256) (:IMM64 . 512) (:IMM1 . 1024) (:BASEINDEX . 2048) (:DISP8 . 4096) (:DISP16 . 8192) (:DISP32 . 16384) (:DISP32S . 32768) (:DISP64 . 65536) (:INOUTPORTREG . 131072) (:SHIFTCOUNT . 262144) (:CONTROL . 524288) (:DEBUG . 1048576) (:TEST . 2097152) (:FLOATREG . 4194304) (:FLOATACC . 8388608) (:SREG2 . 16777216) (:SREG3 . 33554432) (:ACC . 67108864) (:JUMPABSOLUTE . 134217728) (:REGMMX . 268435456) (:REGXMM . 536870912) (:ESSEG . 1073741824) (:INVMEM . 2147483648) (:REG . 15) (:WORDREG . 14) (:IMPLICITREGISTER . 75890688) (:IMM . 1008) (:ENCIMM . 464) (:DISP . 126976) (:ANYMEM . 2147547136) (:LLONGMEM . 2147547136) (:LONGMEM . 2147547136) (:SHORTMEM . 2147547136) (:WORDMEM . 2147547136) (:BYTEMEM . 2147547136) (:LABEL . 4294967296) (:SELF . 8589934592))]
#+end_src
turns into a much more readable
#+begin_src lisp
x86::*x86-operand-type-names* [variable = ((:reg8 . 1) (:reg16 . 2) (:reg32 . 4) (:reg64 . 8) (:imm8 . 16) (:imm8s . 32) (:imm16 . 64) (:imm32 . 128) (:imm32s . 256) (:imm64 . 512) ...)]
#+end_src

** Suppressing documentation errors in =documentation*=

Several implementations throw errors when trying to get documentation
for non-existent method combinations, classes, etc. It's convenient to
suppress these:
#+begin_src lisp
(defmethod gimage:documentation* :around (object &optional doc-type)
(ignore-errors (call-next-method)))
#+end_src

Actually, one can try to write an =:around= method for regular
=documentation=, but this modification is not guaranteed to work on
all implementations.

* Contributing

You can help with any of the [[https://github.com/aartaka/graven-image/issues?q=is%3Aopen+is%3Aissue][open issues]] most are well-described and
split into bite-sized tasks. See .github/CONTIBUTING.md for the
contributing guidelines.