{"id":20977237,"url":"https://github.com/technologicat/pydialect","last_synced_at":"2025-05-14T14:32:05.479Z","repository":{"id":57455845,"uuid":"175375871","full_name":"Technologicat/pydialect","owner":"Technologicat","description":"Build languages on Python.","archived":false,"fork":false,"pushed_at":"2021-05-02T00:29:08.000Z","size":111,"stargazers_count":12,"open_issues_count":3,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-09-15T23:18:35.069Z","etag":null,"topics":["dialects","macropy","metaprogramming","programming-language-development","python","python3","python34","syntactic-macros"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Technologicat.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-03-13T08:15:45.000Z","updated_at":"2024-01-13T23:59:39.000Z","dependencies_parsed_at":"2022-09-05T18:12:30.035Z","dependency_job_id":null,"html_url":"https://github.com/Technologicat/pydialect","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Technologicat%2Fpydialect","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Technologicat%2Fpydialect/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Technologicat%2Fpydialect/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Technologicat%2Fpydialect/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Technologicat","download_url":"https://codeload.github.com/Technologicat/pydialect/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225297832,"owners_count":17452010,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dialects","macropy","metaprogramming","programming-language-development","python","python3","python34","syntactic-macros"],"created_at":"2024-11-19T04:57:42.696Z","updated_at":"2024-11-19T04:57:43.387Z","avatar_url":"https://github.com/Technologicat.png","language":"Python","readme":"# Pydialect: build languages on Python\n\n**NOTE** *April 2021: This project is **obsolete**. This technology now lives on in [`mcpyrate`](https://github.com/Technologicat/mcpyrate), and the example dialects can be found in [`unpythonic`](https://github.com/Technologicat/unpythonic).*\n\n```python\nfrom __lang__ import lispython\n\ndef fact(n):\n    def f(k, acc):\n        if k == 1:\n            return acc\n        f(k - 1, k*acc)\n    f(n, acc=1)\nassert fact(4) == 24\nprint(fact(5000))\n```\n\nPydialect makes Python into a language platform, à la [Racket](https://racket-lang.org/).\nIt provides the plumbing that allows to create, in Python, dialects that compile into Python\nat import time. Pydialect is geared toward creating languages that extend Python\nand look almost like Python, but extend or modify its syntax and/or semantics.\nHence *dialects*.\n\nAs examples, we currently provide the following dialects:\n\n  - [**Lispython**: Python with tail-call optimization (TCO), implicit return, multi-expression lambdas](lispython/)\n  - [**Pytkell**: Python with automatic currying and lazy functions](pytkell/)\n  - [**LisThEll**: Python with prefix syntax and automatic currying](listhell/)\n\nAll three dialects support [unpythonic's](https://github.com/Technologicat/unpythonic)\n``continuations`` block macro (to add ``call/cc`` to the language), but do not enable it automatically.\nLispython aims at production quality; the others are intended just for testing.\n\nPydialect itself is only a lightweight infrastructure hook that makes\nit convenient to define and use dialects. To implement the actual semantics\nfor your dialect (which is where all the interesting things happen), you may\nwant to look at [MacroPy](https://github.com/azazel75/macropy). Examples can be\nfound in [unpythonic](https://github.com/Technologicat/unpythonic); see especially\nthe macros. On packaging a set of semantics into a dialect, look at the example\ndialects; all three are thin wrappers around ``unpythonic``.\n\nNote what Pydialect does is similar to the rejected [PEP 511](https://www.python.org/dev/peps/pep-0511/),\nbut via import hooks, as indeed suggested in the rejection notice. Thus, beside dialects proper,\nit is possible to use Pydialect to hook in a custom AST optimizer, by defining a dialect whose\n``ast_transformer`` is actually an optimizer. For ideas, see [here](http://compileroptimizations.com/).\nSome possibilities are e.g. constant folding, hoisting, if optimization and loop unrolling.\n\n\n### Why dialects?\n\nAn extension to the Python language doesn't need to make it into the Python core,\n*or even be desirable for inclusion* into the Python core, in order to be useful.\n\nBuilding on functions and syntactic macros, customization of the language itself\nis one more tool for the programmer to extract patterns, at a higher level.\nHence, beside language experimentation, such extensions can be used as a\nframework that allows shorter and/or more readable programs.\n\nPydialect places language-creation power in the hands of its users, without the\nneed to go to extreme lengths to hack CPython itself or implement from scratch\na custom language that compiles to Python AST or bytecode.\n\nPydialect dialects compile to Python and are implemented in Python, allowing\nthe rest of the user program to benefit from new versions of Python, mostly\northogonally to the development of any dialect.\n\nAt its simplest, a custom dialect can alleviate the need to spam a combination\nof block macros in every module of a project that uses a macro-based language\nextension, such as ``unpythonic.syntax``. Being named as a dialect, a particular\ncombination of macros becomes instantly recognizable,\nand [DRY](https://en.wikipedia.org/wiki/Don't_repeat_yourself):\nthe dialect definition becomes the only place in the codebase that defines\nthe macro combination to be used by each module in the project.\n\nThe same argument applies to custom builtins: any functions or macros that\nfeel like they \"should be\" part of the language layer, so that they won't\nhave to be explicitly imported in each module where they are used.\n\n\n### Using dialects\n\nPlace a **lang-import** at the start of your module that uses a dialect:\n\n```python\nfrom __lang__ import piethon\n```\n\nRun your program (in this example written in the ``piethon`` dialect)\nthrough the ``pydialect`` bootstrapper instead of ``python3`` directly,\nso that the main program gets imported instead of run directly, to trigger\nthe import hook that performs the dialect processing. (Installing Pydialect\nwill install the bootstrapper.)\n\nAny imported module that has a *lang-import* will be detected, and the appropriate\ndialect module (if and when found) will be invoked. The result is then sent to\nthe macro expander (if MacroPy is installed and the code uses macros at that\npoint), after which the final result is imported normally.\n\nThe lang-import must appear as the first statement of the module; only the\nmodule docstring is allowed to appear before it. This is to make it explicit\nthat **a dialect applies to the whole module**. (Local changes to semantics\nare better represented as a block macro.)\n\nAt import time, the dialect importer replaces the lang-import with an\nassignment that sets the module's ``__lang__`` attribute to the dialect name,\nfor introspection. If a module does not have a ``__lang__`` attribute at\nrun time, then it was not compiled by Pydialect. Note that just like with\nMacroPy, at run time the code is pure Python.\n\nThe lang-import is a construct specific to Pydialect. This ensures that the\nmodule will immediately fail if run under standard Python, because there is\nno actual module named ``__lang__``.\n\nIf you use MacroPy, the Pydialect import hook must be installed at index ``0``\nin ``sys.meta_path``, so that the dialect importer triggers before MacroPy's\nstandard macro expander. The ``pydialect`` bootstrapper takes care of this.\nIf you need to enable Pydialect manually for some reason, the incantation\nto install the hook is ``import dialects.activate``.\n\nThe lang-import syntax was chosen as a close pythonic equivalent to Racket's\n``#lang foo``.\n\n\n### Defining a dialect\n\nIn Pydialect, a dialect is any module that provides one or both of the following\ncallables:\n\n   - ``source_transformer``: source text -\u003e source text\n\n        The **full source code** of the module being imported (*including*\n        the lang-import) is sent to the the source transformer. The data type\n        is whatever the loader's ``get_source`` returns, usually ``str``.\n\n        Source transformers are useful e.g. for defining custom infix\n        operators. For example, the monadic bind syntax ``a \u003e\u003e= b``\n        could be made to transform into the syntax ``a.__mbind__(b)``.\n\n        Although the input is text, in practice a token-based approach is\n        recommended; see stdlib's ``tokenize`` module as a base to work from.\n        (Be sure to untokenize when done, because the next stage expects text.)\n\n        **After the source transformer**, the source text must be valid\n        surface syntax for **standard Python**, i.e. valid input for\n        ``ast.parse``.\n\n   - ``ast_transformer``: ``list`` of AST nodes -\u003e ``list`` of AST nodes\n\n        After the source transformer, but before macro expansion, the full AST\n        of the module being imported (*minus* the module docstring and the\n        lang-import) is sent to this whole-module AST transformer.\n\n        This allows injecting implicit imports to create builtins for the\n        dialect, as well as e.g. lifting the whole module (except the docstring\n        and the code to set ``__lang__``) into a ``with`` block to apply\n        some MacroPy block macro(s) to the whole module.\n\n        **After the AST transformer**, the module is sent to MacroPy for\n        macro expansion (if MacroPy is installed, and the module has macros\n        at that point), and after that, the result is finally imported normally.\n\nThe AST transformer can use MacroPy if it wants, but doesn't have to; this\ndecision is left up to each developer implementing a dialect.\n\nIf you make an AST transformer, and have MacroPy, then see ``dialects.util``,\nwhich can help with the boilerplate task of pasting in the code from the\nuser module (while handling macro-imports correctly in both the dialect\ntemplate and in the user module).\n\n**The name** of a dialect is simply the name of the module or package that\nimplements the dialect. In other words, it's the name that needs to be imported\nto find the transformer functions.\n\nNote that a dotted name in place of the ``xxx`` in ``from __lang__ import xxx``\nis not valid Python syntax, so (currently) **a dialect should be defined in a\ntop-level module** (no dots in the name). Strictly, the dialect finder doesn't\nneed to care about this (though it currently does), but IDEs and tools in\ngeneral are much happier with code that does not contain syntax errors.\n(This allows using standard Python tools with dialects that do not introduce\nany new surface syntax.)\n\nA dialect can be implemented using another dialect, as long as there are no\ndependency loops. *Whenever* a lang-import is detected, the dialect importer\nis invoked (especially, also during the import of a module that defines a\nnew dialect). This allows creating a tower of languages.\n\n\n### Combining existing dialects\n\n*Dangerous things should be difficult to do _by accident_.* --[John Shutt](http://fexpr.blogspot.com/2011/05/dangerous-things-should-be-difficult-to.html)\n\nDue to the potentially unlimited complexity of interactions between language\nfeatures defined by different dialects, there is *by design* no automation for\ncombining dialects. In the general case, this is something that requires human\nintervention.\n\nIf you know (or at least suspect) that two or more dialects are compatible,\nyou can define a new dialect whose ``source_transformer`` and ``ast_transformer``\nsimply chain those of the existing dialects (in the desired order; consider how\nthe macros expand), and then use that dialect.\n\n\n### When to make a dialect\n\nOften explicit is better than implicit. There is however a tipping point with\nregard to complexity, and/or simply length, after which implicit becomes\nbetter.\n\nThis already applies to functions and macros; code in a ``with continuations``\nblock (see macros in [unpythonic](https://github.com/Technologicat/unpythonic))\nis much more readable and maintainable than code manually converted to\ncontinuation-passing style (CPS). There's obviously a tradeoff; as PG reminds\nin [On Lisp](http://paulgraham.com/onlisp.html), each abstraction is another\nentity for the reader to learn and remember, so it must save several times\nits own length to become an overall win.\n\nSo, when to make a dialect depends on how much it will save (in a project or\nacross several), and on the other hand on how important it is to have a shared\ncentral definition that specifies a \"language-level\" common ground for a set of\nuser modules.\n\n\n### Dialect implementation considerations\n\nA dialect can do anything from simply adding some surface syntax (such as a\nmonadic bind operator), to completely changing Python's semantics, e.g. by adding\nautomatic tail-call optimization, continuations, and/or lazy functions. The only\nlimitation is that the desired functionality must be (macro-)expressible in Python.\n\nThe core of a dialect is defined as a set of functions and macros, which typically\nlive inside a library. The role of the dialect module is to package that core\nfunctionality into a whole that can be concisely loaded with a single lang-import.\n\nTypically a dialect implicitly imports its core functions and macros, to make\nthem appear as builtins (in the sense of *defined by default*) to modules that\nuse the dialect.\n\nA dialect may also define non-core functions and macros that live in the same\nlibrary; those essentially comprise *the standard library* of the dialect.\n\nFor example, the ``lispython`` dialect itself is defined by a subset of\n``unpythonic`` and ``unpythonic.syntax``; the rest of the library is available\nto be imported manually; it makes up the standard library of ``lispython``.\n\nFor macros, Pydialect supports MacroPy. Technically, macros are optional,\nso Pydialect's dependency on MacroPy is strictly speaking optional.\nPydialect already defines a hook for a full-module AST transform;\nif that (and/or a source transform) is all you need, there's no need\nto have your dialect depend on MacroPy.\n\nHowever, where MacroPy shines is its infrastructure. It provides a uniform\nsyntax to direct AST transformations to apply to particular expressions or\nblocks in the user code, and it provides a hygienic quasiquote system, which\nis absolutely essential for avoiding inadvertent name conflicts (identifier\ncapture, free variable injection). It's also good at fixing missing source\nlocation info for macro-generated AST nodes, which is extremely useful, since\nin Python the source location info is compulsory for every AST node.\n\nSince you're reading this, you probably already know, but be aware that, unlike\nhow it was envisioned during the *extensible languages* movement in the 1960s-70s,\nlanguage extension is hardly an exercise requiring only *modest amounts of labor\nby unsophisticated users* [[1]](http://fexpr.blogspot.com/2013/12/abstractive-power.html).\nEspecially the interaction between different macros needs a lot of thought,\nand as the number of language features grows, the complexity skyrockets.\n(For an example, look at what hoops other parts of ``unpythonic`` must jump\nthrough to make ``lazify`` happy.) Seams between parts of the user program\nthat use or do not use some particular feature (or a combination of features)\nalso require special attention.\n\nPython is a notoriously irregular language, not to mention a far cry from\nhomoiconic, so it is likely harder to extend than a Lisp. Be prepared for some\nhead-scratching, especially when dealing with function call arguments,\nassignments and the two different kinds of function definitions (``def`` and\n``lambda``, one of which supports the full Python language while the other is\nlimited to the expression sublanguage).\n[Green Tree Snakes - the missing Python AST docs](https://greentreesnakes.readthedocs.io/en/latest/index.html)\nis a highly useful resource here.\n\nBe sure to understand the role of ``ast.Expr`` (the *expression statement*) and\nits implications when working with MacroPy. (E.g. ``ast_literal[tree]`` by itself\nin a block-quasiquote is an ``Expr``, where the value is the ``Subscript``\n``ast_literal[tree]``. This may not be what you want, if you're aiming to\nsplice in a block of statements, but it's unavoidable given Python's AST\nrepresentation.\n\nDialects implemented via macros will mainly require maintenance when Python's\nAST representation changes (if incompatible changes or interesting new features\nhit a relevant part). Large parts of the AST representation have remained\nstable over several of the latest minor releases, or even all of Python 3.\nPerhaps most notably, the handling of function call arguments changed in an\nincompatible way in 3.5, along with introducing MatMult and the async\nmachinery. The other changes between 3.4 (2014) and 3.7 (2018) are just\na couple of new node types.\n\nLong live [language-oriented programming](https://en.wikipedia.org/wiki/Language-oriented_programming),\nand have fun!\n\n\n### Notes\n\n``dialects/importer.py`` is based on ``core/import_hooks.py`` in MacroPy 1.1.0b2,\nand was then heavily customized.\n\nIn the Lisp community, surface syntax transformations are known as *reader macros*\n(although technically it's something done at the parsing step, unrelated to\nsyntactic macros).\n\n**Further reading**:\n\n   - [Hacking Python without hacking Python](http://stupidpythonideas.blogspot.com/2015/06/hacking-python-without-hacking-python.html)\n   - [Operator sectioning for Python](http://stupidpythonideas.blogspot.com/2015/05/operator-sectioning-for-python.html)\n   - [How to use loader and finder objects in Python](http://www.robots.ox.ac.uk/~bradley/blog/2017/12/loader-finder-python.html)\n\n**Ground-up extension efforts** that replace Python's syntax:\n\n   - [Dogelang](https://pyos.github.io/dg/), Python with Haskell-like syntax,\n     compiles to Python bytecode.\n   - [Hy](http://docs.hylang.org/en/stable/), a Lisp-2 that compiles to Python AST.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftechnologicat%2Fpydialect","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftechnologicat%2Fpydialect","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftechnologicat%2Fpydialect/lists"}