{"id":22318632,"url":"https://github.com/scymtym/s-expression-syntax","last_synced_at":"2026-01-05T19:33:17.305Z","repository":{"id":136535836,"uuid":"359378530","full_name":"scymtym/s-expression-syntax","owner":"scymtym","description":"Parse CL syntactic constructs in s-expression form (possibly represented as e.g. CSTs) and construct syntax trees","archived":false,"fork":false,"pushed_at":"2024-11-04T01:28:32.000Z","size":882,"stargazers_count":4,"open_issues_count":17,"forks_count":1,"subscribers_count":4,"default_branch":"builder-protocol","last_synced_at":"2025-01-31T05:47:22.277Z","etag":null,"topics":["ast","common-lisp","cst","parsing","s-expressions"],"latest_commit_sha":null,"homepage":"","language":"Common Lisp","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scymtym.png","metadata":{"files":{"readme":"README.org","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-19T08:08:17.000Z","updated_at":"2024-11-04T01:16:29.000Z","dependencies_parsed_at":"2024-11-03T02:24:38.018Z","dependency_job_id":"f21ca231-6bf7-43f1-9104-e23a76e80f86","html_url":"https://github.com/scymtym/s-expression-syntax","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scymtym%2Fs-expression-syntax","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scymtym%2Fs-expression-syntax/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scymtym%2Fs-expression-syntax/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scymtym%2Fs-expression-syntax/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scymtym","download_url":"https://codeload.github.com/scymtym/s-expression-syntax/tar.gz/refs/heads/builder-protocol","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245585811,"owners_count":20639671,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ast","common-lisp","cst","parsing","s-expressions"],"created_at":"2024-12-03T23:42:05.402Z","updated_at":"2026-01-05T19:33:12.286Z","avatar_url":"https://github.com/scymtym.png","language":"Common Lisp","funding_links":[],"categories":[],"sub_categories":[],"readme":"#+TITLE:    s-expression-syntax README\n#+AUTHOR:   Jan Moringen\n#+EMAIL:    jmoringe@techfak.uni-bielefeld.de\n#+LANGUAGE: en\n\n#+OPTIONS:  toc:t num:nil\n#+SEQ_TODO: TODO STARTED | DONE\n\n#+BEGIN_SRC lisp :exports results :results silent\n  (ql:quickload '(\"s-expression-syntax\"\n                  \"architecture.builder-protocol.print-tree\"))\n#+END_SRC\n\n* STARTED Introduction\n\n  The ~s-expression-syntax~ library provides declarative rules for\n  parsing the various kinds of s-expression syntax used in Common\n  Lisp:\n\n  + Special operators\n\n  + Lambda lists\n\n  + Declarations\n\n  + Type specifiers\n\n  + Standard macros\n\n    + +~cl:loop~+ (work in progress)\n\n  + Feature expressions\n\n  + +Format control strings+ (work in progress)\n\n  Note that this library is not a code walker but it could be used to\n  implement a code walker.\n\n* STARTED Concepts\n\n  Explaining things in more detail requires a little bit of\n  terminology:\n\n  + Expression :: A data structure which /conceptually/\n       consists of atoms and conses (possibly meant for evaluation in\n       which case the expression is a /form/). In contrast to the\n       usual definition, expressions processed by this library may be\n       represented as arbitrary objects which are interpreted as atoms\n       and conses according to rules defined by the client using the\n       library. That said, processing expressions using the usual\n       interpretation of atoms and conses is of course\n       supported. Cons expressions can contain other expressions which\n       are /sub-expressions/ of the respective containing expression.\n\n  + Syntax Description :: An object which describes the syntax of one\n       kind of /expression/. For example, there is a syntax\n       description for the special operator ~cl:let~ which describes\n       (the infinite set) of well-formed ~let~ /expressions/. Besides\n       recognizing well-formed /expressions/ of a certain kind, a\n       /syntax description/ also contains descriptions of the /parts/\n       that can appear within well-formed /expressions/ of that\n       kind. For example, for ~cl:let~, the /parts/ are\n\n       + zero or more bindings, in turn consisting of\n\n         + a variable name\n\n         + an optional initial value form\n\n       + zero or more declarations\n\n       + and zero or more body forms.\n\n       More precisely, a /syntax description/ consists of a name, a\n       list of /parts/ and a parser which consumes /expressions/\n       conforming to the described syntax and produces a parse /result\n       tree/.\n\n  + Part :: An object which describes a sub-structure of\n       the syntax described by a particular /syntax description/. For\n       example, the /syntax description/ for ~cl:let~ has a /part/ for\n       declaration /expressions/ contained in ~let~ /expressions/.\n\n       More precisely, a /part/ consists of a name, a cardinality,\n       which is either one, zero-or-one, one-or-more or zero-or-more,\n       and an evaluation which is roughly \"evaluated\", \"not evaluated\"\n       or \"composed of evaluated and non-evaluated /parts/\".\n\n       For example, ~cl:defun~ has the following /parts/\n\n       | Name                 | Cardinality  | Evaluation                                      |\n       |----------------------+--------------+-------------------------------------------------|\n       | name                 | one          | not evaluated                                   |\n       | lambda list          | one          | composed of evaluated and non-evaluated /parts/ |\n       | documentation string | zero-or-one  | not evaluated                                   |\n       | declaration          | zero-or-more | not evaluated                                   |\n       | body forms           | zero-or-more | evaluated                                       |\n\n       The \"composed of evaluated and non-evaluated /parts/\"\n       evaluation of the lambda list is due to the fact that lambda\n       lists contain unevaluated variable names and lambda list\n       keywords as well as, possibly, evaluated initialization forms\n       for optional and keyword parameters and ~\u0026aux~ variables.\n\n  + Result Tree :: A tree that is result of parsing an\n       input /expression/. Each node in the tree corresponds to a\n       /sub-expression/ of the input /expression/ (multiple nodes may\n       correspond to the same /sub-expression/). A node is\n       characterized by its /kind/, its /initargs/ and its /relations/\n       to other nodes.\n\n       The result tree could possibly be called an Abstract Syntax\n       Tree (AST), but we avoid that term because the result directly\n       reflects the structure of the input /expression/.\n\n       We may call a /result tree/ a /partial result tree/ if it\n       contains unparsed /sub-expressions/ of the input /expression/\n       in some of its leaf nodes.\n\n* STARTED Tutorial\n\n  The most common use of this library probably is turning a given\n  /expression/ into an AST. This process happens in multiple steps\n\n  1. Determine an appropriate /syntax description/ for parsing the\n     /expression/. For example, the /expression/ ~(locally (declare …) 1\n     (+ a b) 3)~ must be parsed using the /syntax description/ for the\n     special operator ~cl:locally~.\n\n  2. Apply the obtained /syntax description/ in conjunction with a parse\n     result builder to obtain a partial (see 3.) AST for the\n     /expression/.\n\n  3. Optionally parse evaluated sub-expressions recursively. In the\n     above example ~(declare …)~ is a sub-expression that is not\n     evaluated while ~1~, ~(+ a b)~ and ~3~ are sub-expressions that\n     are evaluated. The latter are not automatically parsed and thus\n     must be recursively processed in the way described here in order\n     to obtain a fully parsed AST. A complete AST can generally only\n     be produced by consulting an environment as well as interleaving\n     parsing with macroexpansion and is therefore out of scope for\n     this library.\n\n  The following code performs steps 1. and 2. and prints the resulting\n  (partially parsed) AST in a human-readable form. Note how the ~list~\n  builder of the [[https://github.com/scymtym/architecture.builder-protocol][architecture.builder-protocol system]] is passed in the\n  ~parse~ call and later used to destructure the AST node ~node~ by\n  calling the functions ~node-relations~ and ~node-relation~.\n\n  #+NAME: simple-parse\n  #+BEGIN_SRC lisp :exports both :results output\n    (let* ((expression '(defun foo (a \u0026optional (b 2))\n                          (declare (type integer a))\n                          (declare (type integer b))\n                          (format t \"~S\" a)\n                          (list a b)))\n           (syntax     (s-expression-syntax:find-syntax 'defun))\n           ;; Alternatively, determine the appropriate syntax description\n           ;; for EXPRESSION automatically:\n           ;; (syntax     (s-expression-syntax:classify t expression))\n           (builder    'list)\n           (node       (s-expression-syntax:parse builder syntax expression)))\n      (flet ((describe-sub-expression (sub-expression relation-args)\n               (format t \"~2@T-\u003e ~S~%~\n                          ~2@T   evaluation: ~S~%\"\n                       sub-expression (getf relation-args :evaluation))))\n       (loop :for relation    :in (architecture.builder-protocol:node-relations builder node)\n             :for part-name   = (find-symbol (symbol-name (first relation))\n                                             (find-package \"S-EXPRESSION-SYNTAX\"))\n             :for part        = (s-expression-syntax:find-part part-name syntax)\n             :for cardinality = (s-expression-syntax:cardinality part)\n             :for (sub-expression evaluation)\n                = (multiple-value-list (architecture.builder-protocol:node-relation\n                                        builder relation node))\n             :do  (format t \"~A (~A)~%\" part-name cardinality)\n                  (ecase (s-expression-syntax:cardinality part)\n                    ((1) (describe-sub-expression sub-expression evaluation))\n                    ((*) (mapc #'describe-sub-expression sub-expression evaluation))))))\n  #+END_SRC\n\n  Evaluating the code results in the following output which\n  illustrates the four /parts/ of the ~defun~ /expression/: name,\n  lambda-list, declaration and form. The latter two have a cardinality\n  of ~*~, so multiple child nodes may be related to the parent node\n  through the relation in question. In this example, both relations\n  contain two child nodes: two declarations and two body forms.\n\n  #+RESULTS: simple-parse\n  #+begin_example\n  NAME (1)\n    -\u003e (:FUNCTION-NAME NIL :NAME FOO :SOURCE FOO)\n       evaluation: NIL\n  LAMBDA-LIST (1)\n    -\u003e (:ORDINARY-LAMBDA-LIST\n        ((:REQUIRED . *)\n         (((:REQUIRED-PARAMETER\n            ((:NAME . 1)\n             (((:VARIABLE-NAME NIL :NAME A :SOURCE A) :EVALUATION NIL)))\n            :SOURCE A)))\n         (:OPTIONAL . *)\n         (((:OPTIONAL-PARAMETER\n            ((:NAME . 1) (((:VARIABLE-NAME NIL :NAME B :SOURCE B)))\n             (:DEFAULT . 1)\n             (((:UNPARSED NIL :EXPRESSION 2 :CONTEXT :FORM :SOURCE 2) :EVALUATION\n               T)))\n            :SOURCE (B 2))\n           :EVALUATION :COMPOUND)))\n        :SOURCE (A \u0026OPTIONAL (B 2)))\n       evaluation: :COMPOUND\n  DECLARATION (*)\n    -\u003e (:DECLARATION\n        ((:ARGUMENT . *)\n         (((:ATOMIC-TYPE-SPECIFIER\n            ((:NAME . 1) (((:TYPE-NAME NIL :NAME INTEGER :SOURCE INTEGER))))\n            :SOURCE INTEGER))\n          ((:VARIABLE-NAME NIL :NAME A :SOURCE A))))\n        :KIND TYPE :SOURCE (TYPE INTEGER A))\n       evaluation: NIL\n    -\u003e (:DECLARATION\n        ((:ARGUMENT . *)\n         (((:ATOMIC-TYPE-SPECIFIER\n            ((:NAME . 1) (((:TYPE-NAME NIL :NAME INTEGER :SOURCE INTEGER))))\n            :SOURCE INTEGER))\n          ((:VARIABLE-NAME NIL :NAME B :SOURCE B))))\n        :KIND TYPE :SOURCE (TYPE INTEGER B))\n       evaluation: NIL\n  FORM (*)\n    -\u003e (:UNPARSED NIL :EXPRESSION (FORMAT T \"~S\" A) :CONTEXT :FORM :SOURCE\n        (FORMAT T \"~S\" A))\n       evaluation: T\n    -\u003e (:UNPARSED NIL :EXPRESSION (LIST A B) :CONTEXT :FORM :SOURCE (LIST A B))\n       evaluation: T\n  #+end_example\n\n  We can also focus on the overall tree structure and print the\n  (partially parsed) AST as a tree. The following code again uses the\n  =architecture.builder-protocol= system to destructure the AST, this\n  time as part of a generic tree printer.\n\n  #+NAME: tree-parse\n  #+BEGIN_SRC lisp :exports both :results output\n    (let* ((expression '(defun foo (a \u0026optional (b 2))\n                          (declare (type integer a))\n                          (declare (type integer b))\n                          (format t \"~S\" a)\n                          (list a b)))\n           (builder    'list)\n           (node       (s-expression-syntax:parse builder t expression)))\n      (let ((*print-case* :downcase))\n        (architecture.builder-protocol.print-tree:print-tree\n         builder node *standard-output*)))\n  #+END_SRC\n\n  Note the unparsed leafs indicated by the ~unparsed~ node kind.\n\n  #+RESULTS: tree-parse\n  #+begin_example\n  defun\n  │ source: (defun foo (a \u0026optional (b 2))\n  │           (declare (type integer a))\n  │           (declare (type integer b))\n  │           (format t \"~S\" a)\n  │           (list a b))\n  ├─name: function-name\n  │   name: foo\n  │   source: foo\n  ├─lambda-list: ordinary-lambda-list\n  │ │ source: (a \u0026optional (b 2))\n  │ ├─required: required-parameter\n  │ │ │ source: a\n  │ │ └─name: variable-name\n  │ │     name: a\n  │ │     source: a\n  │ └─optional: optional-parameter\n  │   │ source: (b 2)\n  │   ├─name: variable-name\n  │   │   name: b\n  │   │   source: b\n  │   └─default: unparsed\n  │       expression: 2\n  │       context: :form\n  │       source: 2\n  ├─declaration: declaration\n  │ │ kind: type\n  │ │ source: (type integer a)\n  │ ├─argument: atomic-type-specifier\n  │ │ │ source: integer\n  │ │ └─name: type-name\n  │ │     name: integer\n  │ │     source: integer\n  │ └─argument: variable-name\n  │     name: a\n  │     source: a\n  ├─declaration: declaration\n  │ │ kind: type\n  │ │ source: (type integer b)\n  │ ├─argument: atomic-type-specifier\n  │ │ │ source: integer\n  │ │ └─name: type-name\n  │ │     name: integer\n  │ │     source: integer\n  │ └─argument: variable-name\n  │     name: b\n  │     source: b\n  ├─form: unparsed\n  │   expression: (format t \"~S\" a)\n  │   context: :form\n  │   source: (format t \"~S\" a)\n  └─form: unparsed\n      expression: (list a b)\n      context: :form\n      source: (list a b)\n  #+end_example\n\n* STARTED Dictionary\n\n  #+BEGIN_SRC lisp :results none :exports none :session \"doc\"\n    #.(progn\n        #1=(ql:quickload '(:s-expression-syntax :alexandria :split-sequence))\n        '#1#)\n    (defun doc (symbol kind)\n      (let* ((lambda-list (sb-introspect:function-lambda-list symbol))\n             (string      (documentation symbol kind))\n             (lines       (split-sequence:split-sequence #\\Newline string))\n             (trimmed     (mapcar (alexandria:curry #'string-left-trim '(#\\Space)) lines)))\n        (format nil \"~(~A~) ~\u003c~{~A~^ ~}~:@\u003e~2%~{~A~^~%~}\"\n                symbol (list lambda-list) trimmed)))\n  #+END_SRC\n\n** STARTED Syntax Description Protocol\n\n   #+BEGIN_SRC lisp :results value :exports results :session \"doc\"\n     (doc 's-expression-syntax:find-syntax 'function)\n   #+END_SRC\n\n   #+RESULTS:\n   #+begin_example\n   find-syntax NAME \u0026KEY IF-DOES-NOT-EXIST\n\n   Return the syntax description named NAME, if any.\n\n   IF-DOES-NOT-EXIST controls the behavior in case a syntax description\n   named NAME does not exist. The following values are allowed:\n\n   #'ERROR\n\n   Signal an error if a syntax description named NAME does not exist.\n\n   OBJECT\n\n   Return OBJECT if a syntax description named NAME does not exist.\n   #+end_example\n\n   #+BEGIN_SRC lisp :results value :exports results :session \"doc\"\n     (doc '(setf s-expression-syntax:find-syntax) 'function)\n   #+END_SRC\n\n   #+RESULTS:\n   #+begin_example\n   (setf find-syntax) NEW-VALUE NAME \u0026KEY IF-DOES-NOT-EXIST\n\n   Set the syntax description associated with NAME to NEW-VALUE.\n\n   An existing association for NAME, if any, is replaced.\n\n   IF-DOES-NOT-EXISTS is accepted for parity with FIND-SYNTAX but\n   ignored.\n   #+end_example\n\n   #+BEGIN_SRC lisp :results value :exports results :session \"doc\"\n     (doc 's-expression-syntax:ensure-syntax 'function)\n   #+END_SRC\n\n   #+RESULTS:\n   #+begin_example\n   ensure-syntax NAME CLASS \u0026REST INITARGS\n\n   Associate NAME with a syntax description based on CLASS and INITARGS.\n\n   Return the new or updated syntax description object associated with\n   NAME.\n\n   If the database of syntax descriptions already contains a syntax\n   description for NAME, the existing syntax description object is\n   reinitialized with INITARGS.\n\n   If the database of syntax descriptions does not contain a syntax\n   description for NAME, a new association is created by making an\n   instance of CLASS, initializing it with INITARGS and registering the\n   new object for NAME.\n   #+end_example\n\n** STARTED Parser Protocol\n\n   #+BEGIN_SRC lisp :results value :exports results :session \"doc\"\n     (doc 's-expression-syntax:classify 'function)\n   #+END_SRC\n\n   #+RESULTS:\n   #+begin_example\n   classify CLIENT EXPRESSION\n\n   Classify EXPRESSION, possibly according to specialized behavior of CLIENT.\n\n   Return a syntax description object that roughly reflects the kind of\n   EXPRESSION. Note that a precise classification would have to take into\n   account aspects beyond the syntax, such as the environment, to, for\n   example, distinguish function and macro application or variable\n   references and symbol macro applications. It is always possible to\n   find an appropriate syntax description:\n\n   + If EXPRESSION is a special form, this function returns the syntax\n     description for the corresponding special operator.\n\n   + If EXPRESSION is an application of a standard macro, this function\n     returns the syntax description for that macro.\n\n   + If EXPRESSION a list not covered by the above cases, this function\n     returns the syntax description for a generic (that is, function or\n     macro) application. Note that this case also covers invalid\n     applications such as (1 2 3).\n\n   + If EXPRESSION is a symbol but not a keyword, this function returns a\n     syntax description for a variable reference.\n\n   + If EXPRESSION is any object that is not covered by the above cases,\n     this function returns a syntax description for a self-evaluating\n     object.\n   #+end_example\n\n   #+BEGIN_SRC lisp :results value :exports results :session \"doc\"\n     (doc 's-expression-syntax:parse 'function)\n   #+END_SRC\n\n   #+RESULTS:\n   #+begin_example\n   parse CLIENT SYNTAX EXPRESSION\n\n   Parse EXPRESSION according to SYNTAX, possibly specialized to CLIENT.\n\n   SYNTAX is a designator for a syntax description:\n\n   + If SYNTAX is `t', `classify' is applied to CLIENT and EXPRESSION to\n     obtain an appropriate syntax description object.\n\n   + If SYNTAX is any other symbol, `find-syntax' is called to obtain the\n     syntax description named by SYNTAX. An error is signaled if SYNTAX\n     does not name a syntax description.\n\n   + Otherwise SYNTAX must be a syntax description object.\n\n   EXPRESSION is either one of the kinds of expressions that make up\n   Common Lisp programs (such as forms, type specifiers and declarations)\n   or a particular non-standard representation of such expressions which\n   is specific to CLIENT. For example, a client may choose to represent\n   every sub-expression contained in an expression as a standard object\n   in order to store additional information. If CLIENT employs such a\n   non-standard representation, the protocol named by symbols exported\n   from the `s-expression-syntax.expression-grammar' package has to be\n   implemented by defining appropriate methods.\n\n   If EXPRESSION does not conform to the syntax described by SYNTAX, an\n   error of type `invalid-syntax-error' is signaled.\n\n   If EXPRESSION does conform to the syntax described by SYNTAX, a parse\n   result that associates the parts of SYNTAX with sub-expressions of\n   EXPRESSION is returned. The type and structure of the return value\n   depends on CLIENT as the parse result is constructed using the builder\n   protocol with CLIENT as the builder.\n   #+end_example\n\n# Local Variables:\n# eval: (require 'ob-lisp)\n# End:\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscymtym%2Fs-expression-syntax","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscymtym%2Fs-expression-syntax","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscymtym%2Fs-expression-syntax/lists"}