{"id":19741173,"url":"https://github.com/marrow/dsl","last_synced_at":"2025-06-27T14:06:33.320Z","repository":{"id":66821884,"uuid":"88121438","full_name":"marrow/dsl","owner":"marrow","description":"A Pythonic DSL construction engine for import–time code translation.","archived":false,"fork":false,"pushed_at":"2017-05-25T05:08:36.000Z","size":54,"stargazers_count":12,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"develop","last_synced_at":"2025-04-30T06:38:53.574Z","etag":null,"topics":["cpython","dsl","preprocessing","preprocessor","pypy","python","python-2","python-3","text-processing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/marrow.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-04-13T03:31:09.000Z","updated_at":"2024-02-21T18:59:47.000Z","dependencies_parsed_at":"2023-04-23T06:48:08.453Z","dependency_job_id":null,"html_url":"https://github.com/marrow/dsl","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/marrow/dsl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marrow%2Fdsl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marrow%2Fdsl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marrow%2Fdsl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marrow%2Fdsl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/marrow","download_url":"https://codeload.github.com/marrow/dsl/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marrow%2Fdsl/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262270428,"owners_count":23285164,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cpython","dsl","preprocessing","preprocessor","pypy","python","python-2","python-3","text-processing"],"created_at":"2024-11-12T01:25:02.055Z","updated_at":"2025-06-27T14:06:33.298Z","avatar_url":"https://github.com/marrow.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"==========\nmarrow.dsl\n==========\n\n    © 2011-2017 Alice Bevan-McGregor and contributors.\n\n..\n\n    https://github.com/marrow/dsl\n\n..\n\n    |latestversion| |ghtag| |downloads| |masterstatus| |mastercover| |masterreq| |ghwatch| |ghstar|\n\n\nContents\n========\n\n1. `What is this?`_\n\n   1. `What's with the funny names?`_\n   2. `Rationale and Goals`_\n\n2. `Installation`_\n\n   1. `Development Version`_\n\n3. `Domain Specific Languages`_\n\n   1. `Encoding Naming Scheme`_\n   2. `Lines`_\n   3. `Buffers`_\n   4. `Context`_\n   5. `Engine Metadata`_\n   6. `Engine Customization`_\n   7. `Transformation`_\n\n      1. `Block Transformation`_\n      2. `Inline Transformation`_\n      3. `AST Transformation`_\n\n5. `Version History`_\n6. `License`_\n\n\nWhat is this?\n=============\n\nDomain specific languages allow you to write code in ways more optimized to specialized tasks. These can often be\nthought of as interpreters for programming languages other than the one the interpreter is written in, and you may\nalready be familiar with a few such as template engines, or testing frameworks. Marrow DSL is a framework for easily\nconstructing new ones in Python using a preprocessor methodology performing transformation seamlessly at module import\ntime.\n\nWant to write a really fast template engine of your own? `You can do that. \u003chttps://github.com/marrow/cinje\u003e`__\n\nWant to write a testing framework for story driven development? `Totally possible. \u003chttps://gist.github.com/amcgregor/1338661\u003e`__\n\nWant to obfuscate or encrypt your code? `We don't recommend it \u003chttp://s.webcore.io/fVpr/zzinger-best-practices-morpheus.jpg\u003e`__,\nbut yeah, doable.\n\nWant to prank your co-workers and turn their Python into Pascal or C? `We beg you, please don't do this. \u003chttp://s.webcore.io/d41m/165595-your-scientists-were-so-preocc-K8Ym.jpeg\u003e`__\n\n\nWhat's with the funny names?\n----------------------------\n\nYou might notice the base name of the engine is ``GalfiDecoder`` – this was the original internal project name. Marrow\nprojects tend to follow a more... functionally literal... naming scheme in English, and this legacy remains. So what\nin the world is \"galfi\"?\n\nIt's a word from the constructed language `Lojban \u003chttp://www.lojban.org/\u003e`_. A combination of Chinese \"gǎi\", English\n\"alter\", Hindi \"badalanā\", Spanish \"modificar\", Russian \"modificirovatʹ\", and Arabic \"gaiar\". It translates as \"(an\nevent) X modifies/alters/changes/transforms/converts Y into Z\", and is a fairly literal interpretation for the\nmechanism this DSL engine provides, where X is module import, Y is your DSL, and Z is Python code.\n\nSpecific engines, such as `cinje \u003chttps://github.com/marrow/cinje\u003e`__ and korcu will be released using their literal\nnames, though. Lojban root words, or *gismu* are neat: built from global natural languages, descriptive, and they\nscratch the regular expression itch.\n\n\nRationale and Goals\n-------------------\n\nWe find most DSLs (especially template engines) in Python to:\n\n1. Be overly complex, often taking a classical lexer/parser/AST approach to language construction. This can be\n   difficult for developers new to the language (or new to programming) to understand or extend, and poses a hurdle to\n   the understanding of the basic principles. Constructing new ways to write code should be easy and accessible, not\n   difficult and opaque wizardry.\n\n2. Repeatedly solve the same problems in similar ways that could benefit from deduplication between engines. The needs\n   of most engines are similar; these should be fulfilled by a common codebase benefitting many engines.\n\n3. Duplicate functionality such as the import pipeline (e.g. to acquire an invokable object to generate templated\n   text) or bytecode caching layer already present in Python, instead of leveraging these built-in tools.\n\nMarrow DSL takes a simpler approach than most by:\n\n1. Treating the domain-specific code fundamentally as lines of input *text* which can trigger transformations and code\n   generation, reducing lexing/parsing problems to simple string matching and manipulation. This results in a basic DSL\n   framework less than a quarter the size of an average template engine, and engines utilizing Marrow DSL a fraction\n   of that.\n\n2. Ensuring transformation is seamless at module import time, allowing full utilization of Python's own internal\n   bytecode cache as well as the existing package/module discovery and import mechanisms.\n\n3. Allowing the bytecode resulting from translation (as managed by Python itself) to have no dependency at all on the\n   engine that produced it, making the engine (and Marrow DSL) a build time, not production deployment dependency.\n\nDomain-specific languages written using Marrow DSL integrate into general Python codebases seamlessly, and are\ntransformed in a predictable, understandable way that is easy to extend.\n\n\nInstallation\n============\n\nInstalling ``marrow.dsl`` is easy, just execute the following in a terminal::\n\n    pip install marrow.dsl\n\n**Note:** We *strongly* recommend always using a container, virtualization, or sandboxing environment of some kind when\ndeveloping using Python; installing things system-wide is yucky (for a variety of reasons) nine times out of ten.  We\nprefer light-weight `virtualenv \u003chttps://virtualenv.pypa.io/en/latest/virtualenv.html\u003e`__, others prefer solutions as\nrobust as `Vagrant \u003chttp://www.vagrantup.com\u003e`__.\n\nIf you add ``marrow.dsl`` to the ``install_requires`` argument of the call to ``setup()`` in your application's\n``setup.py`` file, the engine will be automatically installed and made available when your own application or\nlibrary is installed. You can alternatively make ``marrow.dsl`` a build-time dependency by declaring it against the\n``setup_requires`` argument instead.\n\nWe recommend \"less than\" version number pinning to ensure there are no unintentional side-effects when updating.  Use\n``marrow.dsl\u003c1.1`` to get all bugfixes for the current release, and ``marrow.dsl\u003c2.0`` to get bugfixes and feature\nupdates while ensuring that large breaking changes are not installed.\n\n\nDevelopment Version\n-------------------\n\n    |developstatus| |developcover| |ghsince| |issuecount| |ghfork|\n\nDevelopment takes place on `GitHub \u003chttps://github.com/\u003e`__ in the\n`marrow/dsl \u003chttps://github.com/marrow/dsl/\u003e`__ project.  Issue tracking, documentation, and downloads\nare provided there.\n\nInstalling the current development version requires `Git \u003chttp://git-scm.com/\u003e`_, a distributed source code management\nsystem.  If you have Git you can run the following to download and *link* the development version into your Python\nruntime::\n\n    git clone https://github.com/marrow/dsl.git\n    (cd dsl; python setup.py develop)\n\nYou can then upgrade to the latest version at any time::\n\n    (cd dsl; git pull; python setup.py develop)\n\nIf you would like to make changes and contribute them back to the project, fork the GitHub project, make your changes,\nand submit a pull request.  This process is beyond the scope of this documentation; for more information see\n`GitHub's documentation \u003chttp://help.github.com/\u003e`_.\n\n\nDomain Specific Languages\n=========================\n\nA Marrow DSL boils down to two things: DSL metadata registration and processing customization, represented as a class\nregistered via ``entry_points`` under the ``marrow.dsl`` namespace, and; one or more transformation classes registered\nunder the ``entry_points`` namespace for your named DSL which are used to inspect, claim, and transform lines of input.\n\nThe mechanism by which transformation is triggered may be somewhat alien: Python unicode decoding hooks for source\nfiles, executed when opening the source file, prior to parsing, compilation, byte code storage, and evaluation during\nimport. To control this magic requires the internal use of Unicode encoding declaration and the ``# [en]coding:``\nmodule encoding declaration to trigger transformation at import time.\n\nPython modules written using a DSL are otherwise just ``.py`` files given a DSL encoding declaration.\n\nIn accordance with `PEP 3120 \u003chttps://www.python.org/dev/peps/pep-3120/\u003e`__, the default encoding of the underlying\ntextual content of all pre-transformation DSLs is UTF-8. Transformers should only operate on native unicode text\nunless additional processing, such as AST analysis, is absolutely required for the operation of the transformer. The\nstandard library includes a vast amount of introspection, parsing, compilation, and other tools prior to needing to\nprocess and regenerate the whole source file from an abstraction. Any DSL whose purpose is the generation of text\nshould similarly default to UTF-8 output.\n\n\nEncoding Naming Scheme\n----------------------\n\nDSLs may have flags and simple options associated with them. Due to limitations on the way Python searches for\nencoding prefixes on source files, the names available are restricted.\n\n1. Within the general name for a specific DSL, any alphanumeric characters (``a-z``, ``0-9``, regardless of case) may\n   be used. This name is parsed early and used to look up the appropriate named metadata ``entry_point`` from the \n   ``marrow.dsl`` namespace. E.g.: ``cinje``\n\n2. Allowed flags must be declared via ``FLAGS`` DSL metadata and are enabled within individual encoding declarations\n   as suffixes on the name, with the same restrictions while allowing hyphens, each prefixed with a period. Multiple\n   may be concatenated and should be lexicographically sorted. E.g. the ``raw`` and ``unsafe`` flags on the ``cinje``\n   encoding: ``cinje.raw.unsafe``\n\n3. Options are identified as hyphen-separated key value pairs. These are kept unambiguous from flags containing\n   hyphens by the explicit declaration of allowed flags in the DSL metadata. Allowed options are defined through\n   assignment of ``__slots__`` explicitly naming options to allocate storage for. (This causes Python to forbid\n   assignment of unknown attributes.) While the value may contain hyphens, the key may not contain any.\n   Numeric-seeming values will be cast to integers automatically during encoding declaration parsing.\n\n\nLines\n-----\n\nLines of code, both input written in the DSL and output Python code, are individually represented by ``Line``\ninstances. Collections of lines are stored in ``Lines`` instances. At all scales tags are used to help identify\nthe lines and collections, represented as sets.\n\n``Line`` defines the content, original line number, scope, and metadata for a single line.  ``Lines`` represents\n\n\n``marrow.dsl.core:Line``\n~~~~~~~~~~~~~~~~~~~~~~~~\n\nThe attributes of a line are:\n\n* ``line`` - The string (unicode text) value of the line.\n\n* ``stripped`` - A whitespace stripped (leading and trailing) version of the line.\n\n* ``number`` - The line number this line originated from in the source, or was triggered by for the purpose of code\n  generation.\n\n* ``scope`` - The Python scope, denoted by indentation level in the resulting code. Leading whitespace in a given line\n  already has the scope-based indentation removed. This means indented blocks in docstrings, manually aligned\n  continuations, and other such situations will have their additional whitespace preserved.\n\n* ``tag`` - An optional set of tags to associate with the line. For example, a built-in tag to identify lines that are\n  manually wrapped and \"continued\" on the next line there is the ``continued`` tag.\n\nEach ``Line`` offers a rich programmers' representation and upon casting to a unicode string will regenerate the line,\nincluding leading indentation. As most lines are constructed from the mutation of an existing line, or based on a\ntriggering line in the case of code generation, two methods are provided to assist:\n\n* ``clone(**kw)`` - Return a new ``Line`` instance as a mutable shallow copy. Any arguments provided will override\n  (replace) the attribute of the same name.\n\n* ``format(*args, **kw)`` - Apply string formatting interpolation (``str.format``)) to the ``line`` attribute, and\n  return a clone of the line with this new value.\n\n\n``marrow.dsl.core:Lines``\n~~~~~~~~~~~~~~~~~~~~~~~~~\n\n\n\nLogical Lines\n~~~~~~~~~~~~~\n\nTBD\n\n* common metadata\n* extended metadata\n* origin tracing\n* continuation\n\n\nBuffers\n-------\n\nTBD\n\n* common metadata\n* context stack\n* reentrant FIFO, push to head mid-iteration\n* named sections\n\n\nContext\n-------\n\n* global metadata\n* reentrant line producer\n* named scopes\n\n\nEngine Metadata\n---------------\n\nTBD\n\n\nEngine Customization\n--------------------\n\nTBD\n\n\nTransformation\n--------------\n\nTransformation is a stack-based, almost coroutine-like streaming process utilizing Python's yield syntax extensively. Individual transformers cooperate to construct the working context as they go, with block transformers manipulating whole lines, and inline transformers manipulating substrings of a line. Additionally, block transformers may be unbuffered, where they may generate one or more lines in response to a line, or buffered, where they act as context managers helping to subdivide the source text into logical sections by constructing \"nested\" (though not really) buffers.\n\n\nBlock Transformation\n~~~~~~~~~~~~~~~~~~~~\n\nTBD\n\n* unbuffered\n* buffered\n\n\nInline Transformation\n~~~~~~~~~~~~~~~~~~~~~\n\nTBD\n\n* delimited interpolation\n\n\nAST Transformation\n~~~~~~~~~~~~~~~~~~\n\nTBD\n\n* buffer context exit triggered\n* post other transformation on the buffer contents\n\n\nVersion History\n===============\n\nVersion 1.0\n-----------\n\n* Initial release.\n\n\nLicense\n=======\n\nMarrow DSL (``marrow.dsl``) has been released under the MIT Open Source license.\n\nThe MIT License\n---------------\n\nCopyright © 2011-2017 Alice Bevan-McGregor and contributors.\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of this software and associated\ndocumentation files (the “Software”), to deal in the Software without restriction, including without limitation the\nrights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit\npersons to whom the Software is furnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all copies or substantial portions of the\nSoftware.\n\nTHE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE\nWARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR\nCOPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR\nOTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n\n.. |ghwatch| image:: https://img.shields.io/github/watchers/marrow/dsl.svg?style=social\u0026label=Watch\n    :target: https://github.com/marrow/dsl/subscription\n    :alt: Subscribe to project activity on Github.\n\n.. |ghstar| image:: https://img.shields.io/github/stars/marrow/dsl.svg?style=social\u0026label=Star\n    :target: https://github.com/marrow/dsl/subscription\n    :alt: Star this project on Github.\n\n.. |ghfork| image:: https://img.shields.io/github/forks/marrow/dsl.svg?style=social\u0026label=Fork\n    :target: https://github.com/marrow/dsl/fork\n    :alt: Fork this project on Github.\n\n.. |masterstatus| image:: http://img.shields.io/travis/marrow/dsl/master.svg?style=flat\n    :target: https://travis-ci.org/marrow/dsl/branches\n    :alt: Release build status.\n\n.. |mastercover| image:: http://img.shields.io/codecov/c/github/marrow/dsl/master.svg?style=flat\n    :target: https://codecov.io/github/marrow/dsl?branch=master\n    :alt: Release test coverage.\n\n.. |masterreq| image:: https://img.shields.io/requires/github/marrow/dsl.svg\n    :target: https://requires.io/github/marrow/dsl/requirements/?branch=master\n    :alt: Status of release dependencies.\n\n.. |developstatus| image:: http://img.shields.io/travis/marrow/dsl/develop.svg?style=flat\n    :target: https://travis-ci.org/marrow/dsl/branches\n    :alt: Development build status.\n\n.. |developcover| image:: http://img.shields.io/codecov/c/github/marrow/dsl/develop.svg?style=flat\n    :target: https://codecov.io/github/marrow/dsl?branch=develop\n    :alt: Development test coverage.\n\n.. |developreq| image:: https://img.shields.io/requires/github/marrow/dsl.svg\n    :target: https://requires.io/github/marrow/dsl/requirements/?branch=develop\n    :alt: Status of development dependencies.\n\n.. |issuecount| image:: http://img.shields.io/github/issues-raw/marrow/dsl.svg?style=flat\n    :target: https://github.com/marrow/dsl/issues\n    :alt: Github Issues\n\n.. |ghsince| image:: https://img.shields.io/github/commits-since/marrow/dsl/1.0.0.svg\n    :target: https://github.com/marrow/dsl/commits/develop\n    :alt: Changes since last release.\n\n.. |ghtag| image:: https://img.shields.io/github/tag/marrow/dsl.svg\n    :target: https://github.com/marrow/dsl/tree/1.0.0\n    :alt: Latest Github tagged release.\n\n.. |latestversion| image:: http://img.shields.io/pypi/v/marrow.dsl.svg?style=flat\n    :target: https://pypi.python.org/pypi/marrow.dsl\n    :alt: Latest released version.\n\n.. |downloads| image:: http://img.shields.io/pypi/dw/marrow.dsl.svg?style=flat\n    :target: https://pypi.python.org/pypi/marrow.dsl\n    :alt: Downloads per week.\n\n.. |cake| image:: http://img.shields.io/badge/cake-lie-1b87fb.svg?style=flat\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarrow%2Fdsl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarrow%2Fdsl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarrow%2Fdsl/lists"}