{"id":15337251,"url":"https://github.com/vy/meta-sexp","last_synced_at":"2026-01-11T10:58:18.137Z","repository":{"id":542041,"uuid":"171940","full_name":"vy/meta-sexp","owner":"vy","description":"A META parser generator using LL(1) grammars with s-expressions.","archived":false,"fork":false,"pushed_at":"2017-10-30T14:39:27.000Z","size":36,"stargazers_count":15,"open_issues_count":0,"forks_count":4,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-12-06T15:22:07.143Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Common Lisp","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2009-04-09T14:24:27.000Z","updated_at":"2024-09-25T22:49:27.000Z","dependencies_parsed_at":"2022-07-07T15:30:21.949Z","dependency_job_id":null,"html_url":"https://github.com/vy/meta-sexp","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vy%2Fmeta-sexp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vy%2Fmeta-sexp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vy%2Fmeta-sexp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vy%2Fmeta-sexp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vy","download_url":"https://codeload.github.com/vy/meta-sexp/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":236721597,"owners_count":19194455,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-01T10:20:18.202Z","updated_at":"2025-10-16T13:30:36.128Z","avatar_url":"https://github.com/vy.png","language":"Common Lisp","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Overview\n\nmeta-sexp is a META parser generator using LL(1) grammars with s-expressions. meta-sexp uses in-memory string vectors, instead of commonly used streams, for efficiently stepping backward and forward through the input. It is tested on SBCL but should be portable to other implementations as well.\n\nmeta-sexp is implemented using sevaral transformation methods. Therefore, besides builtin grammar transformators coming with meta-sexp by default, you are allowed to add your own transformation methods too.\n\nInspired by `src/parser.lisp` of [core-stream project](http://core.gen.tr/).\n\nIdea is based on the META language discussed in [Pragmatic Parsing in Common Lisp](http://home.pipeline.com/~hbaker1/Prag-Parse.html) paper of [Henry G. Baker](http://home.pipeline.com/~hbaker1/home.html).\n\n# Quick Introduction\n\nMost of the time, you'll need to define your own parsers using `CREATE-PARSER-CONTEXT` methods and `DEFRULE`, `DEFRENDERER` macros.\n\n    create-parser-context ((input string) \u0026key start end attachment)\n    create-parser-context ((input string-stream) \u0026key buffer-size start end attachment)\n\n    defrule (name (\u0026rest args) (\u0026optional attachment) \u0026body body)\n    defrenderer (name (\u0026rest args) (\u0026optional attachment) \u0026body body)\n\nIn a rule or renderer body, if supplied, `ATTACHMENT` argument will get bound to `ATTACHMENT` keyword given to `CREATE-PARSER-CONTEXT`.\n\nIn some certain situations, you may also need to use `DEFATOM` too. See `atoms.lisp` for `DEFATOM` examples.\n\nHere is a tiny example:\n\n    (defrule integer? (\u0026aux (sign 1) d (num 0)) ()\n      (:? (:or (:and \"-\" (:assign sign -1))\n               \"+\"))\n      (:+ (:assign d (:type digit?))\n          (:assign num (+ (* num 10)\n\t                  (- (char-code d) #.(char-code #\\0)))))\n      (:return (* sign num)))\n\n    (integer? (create-parser-context \"+123\")) ==\u003e 123\n    (integer? (create-parser-context \"-123\")) ==\u003e -123\n\nHere is another example demonstrating the usage of `META` symbol.\n\n    (defrule in-wonderland? () ()\n      \"META-SEXP\"\n      (progn\n        (format t \"META-SEXP in Wonderland!\")\n        (meta (:type space?)\n  \t    \"in Wonderland!\"))\n      (:return t))\n\n    (in-wonderland?\n     (create-parser-context \"META-SEXP in Wonderland!\"))\n    META-SEXP in Wonderland!\n    ==\u003e T\n\n    (in-wonderland?\n     (create-parser-context \"META-SEXP in Fooland!\"))\n    META-SEXP in Wonderland!\n    ==\u003e NIL\n\nHere's a complete example with renderers and attachments.\n\n    (defrenderer internal-link! (label \u0026optional text) (attachment)\n      (format attachment \"\u003ca href='~a'\u003e~a\u003c/a\u003e\"\n              label (if (empty-char-accum-p text) label text)))\n    \n    (defrule internal-link? (\u0026aux (ref (make-char-accum)) (text (make-char-accum))) ()\n      \"[[\"\n      (:+ (:not (:or \"]]\" (:type (or white-space? newline?))))\n          (:char-push ref))\n      (:? (:* (:type (or white-space? newline?)))\n          (:+ (:not \"]]\")\n  \t    (:char-push text)))\n      \"]]\"\n      (:render internal-link! ref text))\n    \n    (defrule wiki-markup? (\u0026aux c) (attachment)\n      (:* (:or (:rule internal-link?)\n               (:and (:assign c (:read-atom))\n  \t           (write-char c attachment))))\n      (get-output-stream-string attachment))\n\n    (wiki-markup?\n     (create-parser-context\n      \"foo bar [[ref text]] and [[just-ref]] here.\"\n      :attachment (make-string-output-stream)))\n    ==\u003e \"foo bar \u003ca href='ref'\u003etext\u003c/a\u003e and \u003ca href='just-ref'\u003ejust-ref\u003c/a\u003e here.\"\n\nWhat's the role of `ATTACHMENT` slot given to `CREATE-PARSER-CONTEXT` (or specified as a keyword while making an instance of `PARSER-CONTEXT` class)? Think it as a state storage unit between passes to defined rules and renderers. (For instance, in our above example, `ATTACHMENT` used as a common output stream.) Yes, it is possible to let this problem get solved by the programmer via global variables. But this approach yields to another problem: thread safety. Anyway, that was the best that I can come up with; if you have any other ideas, I'd be happy to hear them.\n\nHere is an example introducing a new transformation to the grammar:\n\n    (defun callback (string cursor \u0026rest args)\n      (format t \"Callback at char ~S at position ~D. (Args: ~S)~%\"\n              (elt string cursor) cursor args))\n\n    (defmethod meta-sexp:transform-grammar\n        (ret ctx (in-meta (eql t)) (directive (eql :callback)) \u0026optional args)\n      \"\\(:CALLBACK ARG ...)\n\n    Calls CALLBACK function, where the first two arguments passed to the function\n    are the input string and cursor position, and the rest is ARG.\"\n      `(apply #'callback\n              (meta-sexp::parser-context-data ,ctx)\n              (meta-sexp::parser-context-cursor ,ctx)\n              ,@args))\n\n    (defrule integer-debug? () ()\n      (:or (:rule integer?)\n           (:callback (list :rule :integer-debug?))))\n\n    ; TEST\u003e (integer-debug? (create-parser-context \"xy+123\"))\n    ; Callback at char #\\x at position 0. (Args: (:RULE :INTEGER-DEBUG?))\n    ; NIL\n\n    ; TEST\u003e (integer-debug? (create-parser-context \"+123\"))\n    ; 123\n\n# Available Type Checkers\n\nThese functions (and types) are routines introduced using `DEFATOM` and operates on character codes. In case of need, you can add your own type checkers. (See source for examples.)\n\n    ALNUM? ALPHA? GRAPHIC? ASCII? BIT?\n    DIGIT? EXTENDED? LOWER? NEWLINE?\n    SPACE? TAB? UPPER? WHITE-SPACE?\n\n# Built-In Transformations\n\n    (:ICASE FORM FORM ...)\n\n\u003e Make case-insensitive atom comparison in supplied `FORM`s.\n\n    (:CHECKPOINT FORM)\n\n\u003e Sequentially evaluates supplied forms and if any of them fails, moves cursor back to its start position `:CHECKPOINT` began.\n\n    (:AND FORM FORM ...)\n    (:OR FORM FORM ...)\n    (:NOT FORM)\n\n\u003e Identical to `(NOT FORM)`. (`FORM` is encapsulated within a `:CHECKPOINT` before getting evaluated.)\n\n    (:RETURN VALUE VALUE ...)\n\n\u003e Returns from the rule with supplied `VALUE`s.\n\n    (:RENDER RENDERER ARG ARG ...)\n\n\u003e Calls specified `RENDERER` (that is defined with `DEFRENDERER`) with supplied arguments.\n\n    (:? FORM FORM ...)\n\n\u003e Sequentially evaluates supplied `FORM`s within an `AND` scope and regardless of the return value of `AND`ed `FORM`s, block returns `T`. (Similar to `?` in regular expressions.)\n\n    (:* FORM FORM ...)\n\n\u003e Sequentially evaluates supplied `FORM`s within an `AND` scope until it returns `NIL`. Regardless of the return value of `AND`ed `FORM`s, block returns `T`. (Similar to `*` in regular expressions.)\n\n    (:+ FORM FORM ...)\n\n\u003e Sequentially evaluates supplied `FORM`s within an `AND` scope, and repeats this process till `FORM`s return `NIL`. Scope returns `T` if `FORM`s returned `T` once or more, otherwise returns `NIL`. (Similar to `{1,}` in regular expressions.)\n\n    (:TYPE TYPE-CHECKER)\n    (:TYPE (OR TYPE-CHECKER TYPE-CHECKER ...))\n\n\u003e Checks type of the atom at the current position through supplied function(s).\n\n    (:RULE RULE ARG ARG ...)\n    (:RULE (OR RULE RULE ...) ARG ARG ...)\n\n\u003e Tests input in the current cursor position using specified type/form. If any, supplied arguments will get passed to rule.\n\n    (:ASSIGN VAR FORM)\n    (:ASSIGN (VAR1 VAR2 ...) FORM)\n\n\u003e Assigns returned value of `FORM` to `VAR`, and returns assigned value. (Latter form works similar to `MULTIPLE-VALUE-SETQ`.)\n\n    (:LIST-PUSH ITEM-VAR LIST-ACCUM)\n    (:CHAR-PUSH CHAR-VAR CHAR-ACCUM)\n    (:CHAR-PUSH CHAR-ACCUM)\n\n\u003e Pushes supplied `ITEM-VAR`/`CHAR-VAR` into specified `LIST-ACCUM`/`CHAR-ACCUM`. If `:CHAR-PUSH` is called with only one argument, current character gets read and pushed into supplied accumulator. (You can use `MAKE-LIST-ACCUM` and `MAKE-CHAR-ACCUM` functions to initialize new accumulators. Moreover, you'll probably need `EMPTY-LIST-ACCUM-P` and `EMPTY-CHAR-ACCUM-P` predicates too.)\n\n    (:LIST-RESET LIST-ACCUM)\n    (:CHAR-RESET CHAR-ACCUM)\n\n\u003e Resets supplied accumulators.\n\n    (:EOF)\n\n\u003e Returns true when reached to the end of supplied input data.\n\n    (:READ-ATOM)\n\n\u003e Reads current atom at the cursor position and returns read atom.\n\n    (:DEBUG)\n    (:DEBUG VAR)\n\n\u003e Prints current character and its position in the input data. If `VAR` is specified, prints the value of the `VAR`.\n\nIf a form doesn't start with any of the above keywords, there're three possiblities remaining:\n\n1. This can be a character.\n2. This can be a string. (Will get expanded into an `AND`ed character list with an outermost `:CHECKPOINT`.)\n3. Treat as a custom form. (Will get evaluated as is.)\n\nWhen you're in the third situation, to be able to get your META s-expressions compiled again, use `META` keyword. (See the second example in the Quick Introduction.)\n\n# Introducing New Transformations\n\nEvery transformation process issued by meta-sexp is controlled by `TRANSFORM-GRAMMAR` methods.\n\n    (defgeneric transform-grammar (ctx in-meta directive \u0026optional args)\n      (:documentation \"META grammar transformation methods.\"))\n\nTo introduce a new transformation directive, just create a new `TRANSFORM-GRAMMAR` method with related lambda list specializers. For\ninstance, consider how `:AND` and `:NOT` directive transformations are implemented:\n\n    (defmethod transform-grammar\n        (ctx (in-meta (eql t)) (directive (eql :and)) \u0026optional args)\n      `(and ,@(mapcar #'(lambda (form) (transform-grammar ctx t form))\n                      args)))\n    \n    (defmethod transform-grammar\n        (ctx (in-meta (eql t)) (directive (eql :not)) \u0026optional args)\n      (transform-grammar\n       ctx t :checkpoint\n       `((not ,(transform-grammar ctx t (car args))))))\n\nAlso pay attention how meta-sexp handles unrecognized transformation directives:\n\n    (defmethod transform-grammar (ctx in-meta directive \u0026optional args)\n      \"The most unspecific transformation method.\"\n      (declare (ignore args))\n      (cond\n        ((and in-meta (consp directive) (keywordp (car directive)))\n         (transform-grammar ctx t (car directive) (cdr directive)))\n        ((and (not in-meta) (consp directive) (eql 'meta (car directive)))\n         (transform-grammar ctx t :and (cdr directive)))\n        ((consp directive)\n         (mapcar #'(lambda (form) (transform-grammar ctx nil form))\n                           directive))\n        (t directive)))\n     \nWith similar patterns, you can introduce new transformation directives to meta-sexp.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvy%2Fmeta-sexp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvy%2Fmeta-sexp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvy%2Fmeta-sexp/lists"}