{"id":21025246,"url":"https://github.com/samvv/mage","last_synced_at":"2025-06-22T20:35:17.054Z","repository":{"id":222726201,"uuid":"746763897","full_name":"samvv/mage","owner":"samvv","description":"A work-in-progress lexer/parser generator for a growing number of languages","archived":false,"fork":false,"pushed_at":"2025-02-18T18:19:16.000Z","size":1737,"stargazers_count":3,"open_issues_count":21,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-17T20:37:37.843Z","etag":null,"topics":["lexer-generator","parse-trees","parser-generator","text-analysis"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/samvv.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-01-22T16:21:20.000Z","updated_at":"2025-02-18T18:19:21.000Z","dependencies_parsed_at":"2024-02-15T22:23:39.102Z","dependency_job_id":"d50aa704-b784-4982-ab5a-bc5a70a5b054","html_url":"https://github.com/samvv/mage","commit_stats":null,"previous_names":["samvv/mage"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/samvv/mage","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/samvv%2Fmage","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/samvv%2Fmage/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/samvv%2Fmage/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/samvv%2Fmage/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/samvv","download_url":"https://codeload.github.com/samvv/mage/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/samvv%2Fmage/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261363294,"owners_count":23147535,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["lexer-generator","parse-trees","parser-generator","text-analysis"],"created_at":"2024-11-19T11:31:48.893Z","updated_at":"2025-06-22T20:35:12.041Z","avatar_url":"https://github.com/samvv.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Mage: Text Analysis Made Easy\n=============================\n\nMage is an experimental tool for performing text analysis. It does so by\ngenerating a _lexer_, _parser_ and _parse tree_ for you. Whether it is a piece\nof programming code or some tabular data in a fringe format, Mage has got you\ncovered!\n\n**Features**\n\n - ✅ A simple yet expressive DSL to write your grammars in\n - ✅ Full support for Python typings. Avoid runtime errors while building your language!\n - 🚧 Lots of unit tests to enure your code does what you expect it to do\n - 🚧 An intermediate language to very easily add support to any other programming language\n\n👀 Mage is written in itself. Check out the [generated code][1] of part of our Python generator!\n\n**Implementation Status**\n\n| Feature | Python  | Rust | C  | C++ | JavaScript |\n|---------|---------|------|----|-----|------------|\n| CST     | ✅      | ⏳   | ⏳ | ⏳  | ⏳         |\n| AST     | ⏳      | ⏳   | ⏳ | ⏳  | ⏳         |\n| Lexer   | 🚧      | ⏳   | ⏳ | ⏳  | ⏳         |\n| Parser  | ⏳      | ⏳   | ⏳ | ⏳  | ⏳         |\n| Emitter | ⏳      | ⏳   | ⏳ | ⏳  | ⏳         |\n\n[1]: https://github.com/samvv/mage/blob/main/src/magelang/lang/python/cst.py\n\n## Installation\n\n```\n$ pip3 install --user -U magelang\n```\n\n## Usage\n\nCurrently requires at least Python version 3.12 to run.\n\n### `mage generate \u003clang\u003e \u003cfilename\u003e`\n\nGenerate a parser for the given grammar in a language that you specify.\n\n**Example**\n\n```\nmage generate python foo.mage --prefix foo --out-dir src/foolang\n```\n\n### 🚧 `mage test \u003cfilename..\u003e`\n\n\u003e [!WARNING]\n\u003e\n\u003e This command is under construction.\n\nRun all tests inside the documentation of the given grammar.\n\n### `mage fuzz [filename]`\n\nFuzz the given grammar using a pseudorandom number generator for replayable results.\n\n```\nmage fuzz grammars/magedown.mage\n```\n\n\u003e [!WARNING]\n\u003e\n\u003e If you let `mage fuzz --all` run, Mage may write a lot of data to disk.\n\u003e\n\u003e We recommend creating a temporary file system. A size of 64M should be more\n\u003e than enough. For example:\n\u003e\n\u003e ```sh\n\u003e mkdir -p output/fuzz\n\u003e sudo mount -t tmpfs tmpfs -o size=64m output/fuzz\n\u003e mage fuzz --all\n\u003e ```\n\u003e\n\u003e When you're done run `sudo umount output/fuzz` to unmount it.\n\n## Grammar\n\n### `pub \u003cname\u003e = \u003cexpr\u003e`\n\nDefine a new node or token that must be parsed according the given expression.\n\nYou can use both inline rules and other node rules inside `expr`. When\nreferring to another node, that node will become a field in the node that\nreferred to it. Nodes that have no fields are converted to a special token type\nthat is more efficient to represent.\n\n```\npub var_decl = 'var' name:ident '=' type_expr\n```\n\n### `\u003cname\u003e = \u003cexpr\u003e`\n\nDefine a new inline rule that can be used inside other rules.\n\nAs the name suggests, this type of rule is merely syntactic sugar and gets\ninlined whenever it is referred to inside another rule.\n\n```\ndigits = [0-9]+\n```\n\n### `extern \u003cname\u003e`\n\nDefines a new parsing rule that is defined somewhere else, possibly in a different language.\n\n### `extern token \u003cname\u003e`\n\nDefines a new lexing rule that is defined somewhere else, possibly in a different language.\n\n### `pub token \u003cname\u003e = \u003cexpr\u003e`\n\nLike `pub \u003cname\u003e = \u003cexpr\u003e` but forces the rule to be a token.\n\nMage will show an error when the rule could not be converted to a token rule.\nThis usually means that the rule references another rule that is only `pub`.\n\n```\npub token float\n  = digits? '.' digits\n```\n\n### `pub token \u003cname\u003e -\u003e \u003ctype_expr\u003e = \u003cexpr\u003e`\n\nLike `pub token \u003cname\u003e = \u003cexpr\u003e` but forces the value inside the token to be of\nthe specific type defined by `type_expr`.\n\n```\npub token int -\u003e Integer\n  = digits\n```\n\nThe conversion from the value to the type depends on which type is actually used.\nFor example, the following table is used when targeting Python:\n\n| Mage Type | Python Type | Code           |\n|-----------|-------------|----------------|\n| Integer   | `int`       | `int(value)`   |\n| Float     | `float`     | `float(value)` |\n\n### `expr1 expr2`\n\nFirst parse `expr1` and continue to parse `expr2` immediately after it.\n\n```\npub two_column_csv_line\n  = text ',' text '\\n'\n```\n\n### `expr1 | expr2`\n\nFirst try to parse `expr1`. If that fails, try to parse `expr2`. If none of the\nexpressions matched, the parser fails.\n\n```\npub declaration\n  = function_declaration\n  | let_declaration\n  | const_declaration\n```\n\n### `expr?`\n\nParse or skip the given expression, depending on whether the expression can be\nparsed.\n\n```\npub singleton_or_pair\n  = value (',' value)?\n```\n\n### `expr*`\n\nParse the given expression as much as possible.\n\n```\nskip = (multiline_comment | whitespace)*\n```\n\n### `expr+`\n\nParse the given expression one or more times.\n\nFor example, in Python, there must always be at least one statement in the body of a class or function:\n\n```\nbody = stmt+\n```\n\n### `\\expr`\n\nEscape an expression by making it hidden. The expression will be parsed, but\nnot be visible in the resulting CST/AST.\n\n### `expr{n,m}`\n\nParse the expression at least `n` times and at most `m` times.\n\n```\nunicode_char = 'U+' hex_digit{4,4}\n```\n\n### `@keyword`\n\nTreat the given rule as being a potential source for keywords.\n\nString literals matching this rule will get the special `_keyword`-suffix\nduring transformation. The lexer will also take into account that the rule\nconflicts with keywords and generate code accordingly.\n\n```\n@keyword\npub token ident\n  = [a-zA-Z_] [a-zA-Z_0-9]*\n```\n\n### `@skip`\n\nRegister the chosen rule as a special rule that the lexer uses to lex 'gibberish'.\n\nThe rule will still be available in other rules, e.g. when `@noskip` was added.\n\n```\n@skip\nwhitespace = [\\n\\r\\t ]*\n```\n\n### 🚧 `@noskip`\n\n\u003e [!WARNING]\n\u003e\n\u003e This decorator is under construction.\n\nDisable automatic injection of the `@skip` rule for the chosen rule.\n\nThis can be useful for e.g. parsing indentation in a context where whitespace\nis normally discarded.\n\n```\n@skip\n__ = [\\n\\r\\t ]*\n\n@noskip\npub body\n  = ':' __ stmt\n  | ':' \\indent stmt* \\dedent\n```\n\n### `@wrap`\n\nAdding this decorator to a rule ensures that a real CST node is emitted for\nthat rule, instead of possibly a variant.\n\nThis decorator makes the CST heavier, but this might be warranted in the name\nof robustness and forward compatibility. Use this decorator if you plan to add\nmore fields to the rule.\n\n```\n@wrap\npub lit_expr\n   = literal:(string | integer | boolean)\n```\n\n### `keyword`\n\nA special rule that matches **any keyword present in the grammar**.\n\nThe generated CST will contain predicates to check for a keyword:\n\n```py\nprint_bold = False\nif is_py_keyword(token):\n    print_bold = True\n```\n\n### `token`\n\nA rule that matches **any token in the grammar**.\n\n```\npub macro_call\n  = name:ident '{' token* '}'\n```\n\n### `node`\n\nA special rule that matches **any parseable node in the grammar**, excluding tokens.\n\n### `syntax`\n\nA special rule that matches **any rule in the grammar**, including tokens.\n\n## Python API\n\nThis section documents the API that is generated by taking a Mage grammar as\ninput and specifying `python` as the output language.\n\nIn what follows, `Node` is the name of an arbitrary CST node (such as\n`PyReturnStmt` or `MageRepeatExpr`) and `foo` and `bar` are the name of fields\nof such a node. Examples of field names are `expr`, `return_keyword`, `min`,\n`max,`, and so on.\n\n### `Node(...)`\n\nConstruct a node with the fields specified in the `...` part of the expression.\n\nFirst go all elements that are required, i.e. they weren't suffixed with `?` or\n`*` in the grammar or something similar. They may be specified as positional\narguments or as keyword.\n\nNext are all optional arguments. They **must** be specified as keyword\narguments. When omitted, the corresponding fields are either set to `None` or a\nnew empty token/node is created.\n\n#### Examples\n\n**Creating a new CST node by providing positional arguments for required fields:**\n```py\nPyInfixExpr(\n    PyNamedExpr('value'),\n    PyIsKeyword(),\n    PyNamedExpr('None')\n)\n```\n\n**The same example but now with keyword arguments:**\n```py\nPyInfixExpr(\n    left=PyNamedExpr('value'),\n    op=PyIsKeyword(),\n    right=PyNamedExpr('None')\n)\n```\n\n**Omitting fields that are trivial to construct:**\n```py\n# Note that `return_keyword` is not specified\nstmt = PyReturnStmt(expr=PyConstExpr(42))\n\n# stmt.return_keyword was automatically created\nassert(isinstance(stmt.return_keyword, ReturnKeyword()))\n```\n\n### `Node.count_foos() -\u003e int`\n\nThis member is generated when there was a repetition in field `foo` such\nas the Mage expression `'.'+`\n\nIt returns the amount of elements that are actually present in the CST node.\n\n## FAQ\n\n### What is a CST, AST, visitor, and so on?\n\nA **CST** is a collection of structures and enumerations that completely\nrepresent the source code that needs to be parsed/emitted.\n\nAn **AST** is an abstract representation of the CST. Mage can automatically\nderive a good AST from a CST.\n\nA **visitor** is (usually) a function that traverses the AST/CST in a\nparticular way. It is useful for various things, such as code analysis and\nevaluation.\n\nA **rewriter** is similar to a visitor in that it traverses that AST/CST but\nalso creates new nodes during this traversal.\n\nA **lexer** or scanner is at it core a program that splits the input stream\ninto separate _tokens_ that are easy to digest by the parser.\n\nA **parser** converts a stream of tokens in AST/CST nodes. What parts of the\ninput stream are converted to which nodes usually depends on how the parser is invoked.\n\n### How do I assign a list of nodes to another node in Python without type errors?\n\nThis is probably due to [this feature](https://mypy.readthedocs.io/en/stable/common_issues.html#invariance-vs-covariance)\nin the Python type checker, which prevents subclasses from being assigned to a more general type.\n\nFor small lists, we recommend making a copy of the list, like so:\n\n```py\ndefn = PyFuncDef(body=list([ ... ]))\n```\n\nSee also [this issue](https://github.com/microsoft/pyright/issues/130) in the Pyright repository.\n\n## Contributing\n\nRun the following command in a terminal to link the `mage` command to your checkout:\n\n```\npip3 install -e '.[dev]'\n```\n\n## License\n\nThis code is generously licensed under the MIT license.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsamvv%2Fmage","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsamvv%2Fmage","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsamvv%2Fmage/lists"}