https://github.com/mbdevpl/horast
Human-oriented abstract syntax tree (AST) parser/unparser for Python 3 that doesn't discard comments.
https://github.com/mbdevpl/horast
abstract-syntax-tree comments parser unparsing
Last synced: 5 months ago
JSON representation
Human-oriented abstract syntax tree (AST) parser/unparser for Python 3 that doesn't discard comments.
- Host: GitHub
- URL: https://github.com/mbdevpl/horast
- Owner: mbdevpl
- License: apache-2.0
- Created: 2017-08-25T09:10:26.000Z (almost 8 years ago)
- Default Branch: master
- Last Pushed: 2019-10-29T13:29:59.000Z (over 5 years ago)
- Last Synced: 2024-11-27T21:08:10.978Z (6 months ago)
- Topics: abstract-syntax-tree, comments, parser, unparsing
- Language: Python
- Homepage:
- Size: 112 KB
- Stars: 16
- Watchers: 5
- Forks: 6
- Open Issues: 4
-
Metadata Files:
- Readme: README.rst
- License: LICENSE
Awesome Lists containing this project
README
.. role:: python(code)
:language: python
horast
======
Human-oriented abstract syntax tree (AST) parser/unparser for Python 3 that doesn't discard comments.
.. image:: https://img.shields.io/pypi/v/horast.svg
:target: https://pypi.org/project/horast
:alt: package version from PyPI
.. image:: https://travis-ci.org/mbdevpl/horast.svg?branch=master
:target: https://travis-ci.org/mbdevpl/horast
:alt: build status from Travis CI
.. image:: https://ci.appveyor.com/api/projects/status/github/mbdevpl/horast?svg=true
:target: https://ci.appveyor.com/project/mbdevpl/horast
:alt: build status from AppVeyor
.. image:: https://api.codacy.com/project/badge/Grade/33195093bb1b448bb9a5368b3507d615
:target: https://www.codacy.com/app/mbdevpl/horast
:alt: grade from Codacy
.. image:: https://codecov.io/gh/mbdevpl/horast/branch/master/graph/badge.svg
:target: https://codecov.io/gh/mbdevpl/horast
:alt: test coverage from Codecov
.. image:: https://img.shields.io/github/license/mbdevpl/horast.svg
:target: https://github.com/mbdevpl/horast/blob/master/NOTICE
:alt: license
This package provides new AST node types (Comment and Directive) which inherit from nodes in typed_ast.ast3 module.
Additionally, it provides implementation of parser and unparser for the extended AST allowing
straightforward readable code generation.
Simple example of how to use this package:
.. code:: python
from horast import parse, unparse
tree = parse("""a = 1 # a equals one after this""")
print(unparse(tree))
# this will print the code with original comment
More examples in ``_.
technical details
-----------------
Parser is based on built-in tokenize module, as well as community packages asttokens and typed_ast.
Unparser is essentially an extension of Unparser class from static_typing package.
Nodes provided and handled by horast are listed below.
Comment
~~~~~~~
Full line as well as end-of-line comments are parsed/unparsed correctly when they are outside
of multi-line expressions.
Currently, handling of comments within multi-line expressions is implemented only partially.
BlockComment
~~~~~~~~~~~~
**Not implemented yet.**
This node type is intended to store consecutive full-line comments in a
single AST node, therefore simplifying handling of large blocks of comments.
Directive
~~~~~~~~~
Since Python doesn't offer a direct way to convey directives in code,
comments beginning with any of the following prefixes will not be classified
as typical comments, but as directives:
* if
* else
* endif
* def
* undef
* ifdef
* ifndef
However in Python code they still remain as usual comments.
For example, the comments in the following code will all be classified as directives.
.. code:: python
#ifdef DEBUG
debugging = True
#else
debugging = False
#fi
However, when executed as Python, ``debugging`` will always end up ``False``
because directives are preserved as usual comments in Python and therefore
they are ignored.
Therefore, the ``Directive`` node is not meant to enable preprocessing of
Python, at least for now.
Note: the prefix is checked exactly. See the following example:
.. code:: python
#if something
# if some other thing
The comment in the first line will become ``Directive`` object, while the one
on the second like will become ``Comment`` object.
Currently, this node is meant to work towards AST compatibility between
Python and other languages, to aide code generation from Python AST into code
in other languages -- one such use case is
`*transpyle* project `_.
Pragma
~~~~~~
``Pragma`` nodes follow the same general rules as ``Directive`` nodes, i.e. they are
stored in Python code as comments, but a comment will be classified as a pragma
when it's prefixed with a predefined prefix:
* ``" pragma:"``
Additionally, two subclasses of Pragma are defined in horast, each with its
own prefix that builds upon the generic pragma prefix:
* ``OpenMpPragma`` class defines prefix ``" pragma: omp"`` and stores OpenMP pragmas.
* ``OpenAccPragma`` class defines prefix ``" pragma: acc"`` and stores OpenACC pragmas.
A code snippet below contains all 3 pragma types.
.. code:: python
# pragma: once
use_openmp = True
use_openacc = True
...
a, b = np.ndarray(...)
c = np.zeros(...)
# pragma: acc parallel copyin(a,b) copyout(c)
# pragma: acc loop gang
for y in range(ymax): # type: np.int32
# pragma: acc loop worker
for i in range(imax): # type: np.int32
# pragma: acc loop vector reduction(+: c[y][x])
for x in range(xmax): # type: np.int32
c[y, x] += a[y, i] * b[i, x]
# pragma: acc end parallel
...
# pragma: omp parallel do
for i in range(input_data.size): # type: int
# here we compute spam spam spam
heavy_compute(input_data[i])
...
And thus, in the example above:
* all comments starting with ``" pragma: omp"`` become ``OpenMpPragma`` objects,
* all comments starting with ``" pragma: acc"`` become ``OpenAccPragma`` objects,
* all other comments starting with ``" pragma:"`` become ``Pragma`` objects,
* type comments are ignored, and
* all other comments become ``Comment`` objects.
Additionally, horast module provides an extensible infrastructure to define
custom ``Pragma`` subclasses, enabling user to define their own pragmas for
experimentation. The provided OpenMP and OpenACC pragma definitions serve
as examples of how to use this feature.
Include
~~~~~~~
Similarly to how pragmas are handled, if a comment begins with ``" include:"``
prefix, it will be classified as a special kind of include directive.
Again, this will be preserved as comment in Python code, but it's useful
for enhancing syntactic compatibility between Python and other, especially
statically compiled languages.
requirements
------------
CPython 3.5 or later.
Python libraries as specified in ``_.
Building and running tests additionally requires packages listed in ``_.