{"id":13468936,"url":"https://github.com/zanellia/prometeo","last_synced_at":"2025-05-16T18:08:57.915Z","repository":{"id":40787929,"uuid":"115460100","full_name":"zanellia/prometeo","owner":"zanellia","description":"An experimental Python-to-C transpiler and domain specific language for embedded high-performance computing","archived":false,"fork":false,"pushed_at":"2022-05-19T16:32:39.000Z","size":2024,"stargazers_count":633,"open_issues_count":7,"forks_count":33,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-05-10T15:54:02.392Z","etag":null,"topics":["c","compiler","domain-specific-language","embedded-systems","high-performance-computing","hpc","python","python-to-c","source-to-source","static-analysis","static-typing","transcompiler","transpiler"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zanellia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-12-26T22:45:53.000Z","updated_at":"2025-04-12T10:32:29.000Z","dependencies_parsed_at":"2022-07-11T05:45:59.097Z","dependency_job_id":null,"html_url":"https://github.com/zanellia/prometeo","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zanellia%2Fprometeo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zanellia%2Fprometeo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zanellia%2Fprometeo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zanellia%2Fprometeo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zanellia","download_url":"https://codeload.github.com/zanellia/prometeo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254582907,"owners_count":22095518,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["c","compiler","domain-specific-language","embedded-systems","high-performance-computing","hpc","python","python-to-c","source-to-source","static-analysis","static-typing","transcompiler","transpiler"],"created_at":"2024-07-31T15:01:22.245Z","updated_at":"2025-05-16T18:08:57.886Z","avatar_url":"https://github.com/zanellia.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"\u003cimg src=\"https://github.com/zanellia/prometeo/blob/master/logo/logo.png\" align=\"left\"\n     alt=\"prometeo logo by Andrea Zanelli\" width=\"80\" height=\"120\"\u003e\n\n![Travis Status](https://travis-ci.org/zanellia/prometeo.svg?branch=master) [![PyPI version fury.io](https://badge.fury.io/py/prometeo-dsl.svg)](https://pypi.python.org/pypi/ansicolortags/) [![Documentation Status](https://readthedocs.org/projects/prometeo/badge/?version=latest)](https://prometeo.readthedocs.io/en/latest/?badge=latest)\n\n\nThis is prometeo, an experimental modeling tool for embedded high-performance computing. prometeo provides a \ndomain specific language (DSL) based on a subset of the Python language that allows \none to conveniently write scientific computing programs in a high-level language (Python itself) that can be transpiled\nto high-performance self-contained C code easily deployable on embedded devices.\n\n### features\n- __Python compatible syntax__ : prometeo is a DSL embedded into the Python language. \nprometeo programs can be executed from the Python interpreter.\n- __efficient__ : prometeo programs transpile to high-performance C code.\n- __statically typed__ : prometeo uses Python's native type hints to strictly enforce static typing.\n- __deterministic memory usage__ : a specific program structure is required and enforced through static analysis. In this way\nprometeo transpiled programs have a guaranteed maximum heap usage.\n- __fast memory management__ : thanks to its static analysis, prometeo can avoid allocating\nand garbage-collecting memory, resulting in faster and safer execution.\n- __self-contained and embeddable__ : unlike other similar tools and languages, prometeo targets specifically embedded applications and programs \nwritten in prometeo transpile to self-contained C code that does not require linking against \nthe Python run-time library.\n\n### documentation (preliminary)\nprometeo's documentation can be found on Read the Docs at [https://prometeo.readthedocs.io/en/latest/index.html](https://prometeo.readthedocs.io/en/latest/index.html).\n      \n### hello world!\n\nA simple hello world example that shows how to either run a trivial prometeo program from Python or transpile it to C, build it and run it can be found [here](https://github.com/zanellia/prometeo/blob/master/examples/helloworld/helloworld.py). The output shows the outcome of the heap usage analysis and the execution time (in this case there is not much to see :p).\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/zanellia/prometeo/blob/master/gifs/helloworld.gif\" width=\"100%\" \u003e\n\u003c/p\u003e\n\n\n### performance\nSince prometeo programs transpile to pure C code that calls the high performance linear algebra library BLASFEO (publication: [https://arxiv.org/abs/1704.02457](https://arxiv.org/abs/1704.02457), code: [https://github.com/giaf/blasfeo](https://github.com/giaf/blasfeo)), execution time can be comparable to hand-written high-performance code. The figure below shows a comparison of the CPU time necessary to carry out a Riccati factorization using highly optimized hand-written C code with calls to BLASFEO and the ones obtained with prometeo transpiled code from [this example](https://github.com/zanellia/prometeo/blob/master/examples/riccati_example/riccati_mass_spring.py). The computation times obtained with NumPy and Julia are added too for comparison - notice however that these last two implementations of the Riccati factorization are **not as easily embeddable** as the C code generated by prometeo and the hand-coded C implementation. All the benchmarks have been run on a Dell XPS-9360 equipped with an i7-7560U CPU running at 2.30 GHz (to avoid frequency fluctuations due to thermal throttling). \n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/zanellia/prometeo/blob/master/benchmarks/riccati_benchmark.png\" width=\"100%\" \u003e\n\u003c/p\u003e\n\nMoreover, prometeo can largely outperform state-of-the-art Python compilers such as Nuitka. The table below shows the CPU times obtained on a Fibonacci benchmark.\n\n|    parser/compiler   | CPU time [s] |\n|:--------------------:|--------------|\n| Python 3.7 (CPython) |    11.787    |\n|        Nuitka        |    10.039    |\n|\tPyPy           |     1.78     |\n|       prometeo       |     0.657    |\n\n\n### PyPI installation\n\nprometeo can be installed through PyPI with `pip install prometeo-dsl`. Notice that, since prometeo makes extensive use of [type hints](https://docs.python.org/3.6/library/typing.html) to equip Python code with static typing information, the minimum Python version required is 3.6.\n\n### manual installation\nIf you want to install prometeo building the sources on your local machine you can proceed as follows:\n\n- Run `git submodule update --init` to clone the submodules.\n- Run `make install_shared` from `\u003cprometeo_root\u003e/prometeo/cpmt` to compile and install the shared library associated with the C backend. Notice that the default installation path is `\u003cprometeo_root\u003e/prometeo/cpmt/install`.\n- You need Python 3.6. or later.\n- Optional: to keep things clean you can setup a virtual environment with `virtualenv --python=\u003cpath_to_python3.6\u003e \u003cpath_to_new_virtualenv\u003e`.\n- Run `pip install -e .` from `\u003cprometeo_root\u003e` to install the Python package.\n\nFinally, you can run the examples in `\u003croot\u003e/examples` with `pmt \u003cexample_name\u003e.py --cgen=\u003cTrue/False\u003e`, where the `--cgen` flag determines whether the code is executed by the Python interpreter or C code is generated compiled and run.\n\n### a simple example\n\nThe Python code ([`examples/simple_example/simple_example.py`](https://github.com/zanellia/prometeo/blob/master/examples/simple_example/simple_example.py))\n```python\nfrom prometeo import *\n\nn : dims = 10\n\ndef main() -\u003e int:\n\n    A: pmat = pmat(n, n)\n    for i in range(10):\n        for j in range(10):\n            A[i, j] = 1.0\n\n    B: pmat = pmat(n, n)\n    for i in range(10):\n        B[0, i] = 2.0\n\n    C: pmat = pmat(n, n)\n    C = A * B\n    pmat_print(C)\n    return 0\n```\ncan be run by the standard Python interpreter (version \u003e3.6 required) and it \nwill perform the described linear algebra operations using the command `pmt simple_example.py --cgen=False`. \nAt the same time, the code can be parsed by prometeo and its abstract syntax tree (AST) analyzed in order\nto generate the following high-performance C code:\n```c\n#include \"stdlib.h\"\n#include \"simple_example.h\"\nvoid * ___c_pmt_8_heap;\nvoid * ___c_pmt_64_heap;\nvoid * ___c_pmt_8_heap_head;\nvoid * ___c_pmt_64_heap_head;\n\n#include \"prometeo.h\"\nint main() {\n    ___c_pmt_8_heap = malloc(10000); \n    ___c_pmt_8_heap_head = ___c_pmt_8_heap;\n    char * pmem_ptr = (char *)___c_pmt_8_heap;\n    align_char_to(8, \u0026pmem_ptr);\n    ___c_pmt_8_heap = pmem_ptr;\n    ___c_pmt_64_heap = malloc(1000000);\n    ___c_pmt_64_heap_head = ___c_pmt_64_heap;\n    pmem_ptr = (char *)___c_pmt_64_heap;\n    align_char_to(64, \u0026pmem_ptr);\n    ___c_pmt_64_heap = pmem_ptr;\n\tvoid *callee_pmt_8_heap = ___c_pmt_8_heap;\n\tvoid *callee_pmt_64_heap = ___c_pmt_64_heap;\n\n    struct pmat * A = c_pmt_create_pmat(n, n);\n    for(int i = 0; i \u003c 10; i++) {\n        for(int j = 0; j \u003c 10; j++) {\n            c_pmt_pmat_set_el(A, i, j, 1.0);\n    }\n\n    }\n\n    struct pmat * B = c_pmt_create_pmat(n, n);\n    for(int i = 0; i \u003c 10; i++) {\n        c_pmt_pmat_set_el(B, 0, i, 2.0);\n    }\n\n    struct pmat * C = c_pmt_create_pmat(n, n);\n    c_pmt_pmat_fill(C, 0.0);\n    c_pmt_gemm_nn(A, B, C, C);\n    c_pmt_pmat_print(C);\n\t___c_pmt_8_heap = callee_pmt_8_heap;\n\t___c_pmt_64_heap = callee_pmt_64_heap;\n\n\tfree(___c_pmt_8_heap_head);\n\tfree(___c_pmt_64_heap_head);\n\treturn 0;\n}\n```\nwhich relies on the high-performance linear algebra package BLASFEO. The generated code will be readily compiled and run with when running `pmt simple_example.py --cgen=True`.\n\n### how does it work?\n\nAlthough translating a program written in a language into another with\na comparable level of abstraction can be significantly easier than\ntranslating to one with a very different level of abstraction\n(especially if the target language is of much lower level),\ntranslating Python programs into C programs still involves a\nconsiderable abstraction gap it is not an easy task\nin general. Loosely speaking, the challenge lies in the necessity to\nreimplement features that are natively supported by the source\nlanguage in the target language. In particular, when translating\nPython to C, the difficulty comes both from the different level of\nabstraction of the two languages and from the fact that the source and\ntarget language are of two very different types: Python is an\n*interpreted*, *duck-typed* and\n*garbage-collected* language and C is a\n*compiled* and *statically typed* language.\n\nThe task of transpiling Python to C becomes even more challenging if we add the\nconstraint that the generated C code must be efficient (even for small\nto medium scale computations) and deployable\non embedded hardware. In fact these two requirements directly imply\nthat the generated code cannot make use of: *i)* sophisticated runtime\nlibraries, e.g., the Python runtime library, which are generally not available on\nembedded hardware *ii)* dynamic memory allocation that would make the execution\nslow and unreliable (exception made for memory that is allocated in a\nsetup phase and whose size is known a priori). \n\nSince source-to-source code transformation, or transpilation, and in\nparticular transpilation of Python code into C code is not an\nunexplored realm, in the following, we mention a few existing projects\nthat address it. In doing so, we highlight where and how they do not satisfy one of\nthe two requirements outlined above, namely (small scale) efficiency\nand embeddability. \n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/zanellia/prometeo/blob/master/figures/prometeo_crop.png\" width=\"50%\" \u003e\n\u003c/p\u003e\n     \n### related work\nSeveral software packages exist that address Python-to-C translation in\nvarious forms. \n\nIn the context of high-performance computing,\n*Numba* is a just-in-time compiler for numerical\nfunctions written in\nPython. As such, its aim is to convert properly annotated Python\nfunctions, not entire programs, into high-performance LLVM code such \nthat their execution can be sped\nup. *Numba* uses an internal representation of the code to be\ntranslated and performs a (potentially partial) type inference on the\nvariables involved in order to generate LLVM code that can be called\neither from Python or from C/C++. In some cases, namely the\nones where a complete type inference can\nbe carried out successfully, code that does not rely on the C API can be\ngenerated (using the *nopython* flag). However, the emitted LLVM code\nwould still rely on *Numpy* for BLAS and LAPACK  operations.  \n\n*Nuitka* is a source-to-source compiler that can translate\nevery Python construct into C code that links against the*libpython* library and it is therefore able to transpile a\nlarge class of Python programs. In order to do so, it relies on the fact\nthat one of the most used implementations of the Python language,\nnamely *CPython*, is written in C. In fact, *Nuitka*\ngenerates C code that contains calls to *CPython* that would\nnormally be carried out by the Python parser. Despite its\nattractive and general transpilation approach, it cannot be easily\ndeployed on embedded hardware due to its intrinsic dependency on\n*libpython*. At the same time, since it maps rather closely\nPython constructs to their *CPython* implementation, a number\nof performance issues can be expected when it comes to small to medium\nscale high-performance computing. This is particularly due to the fact\nthat operations associated with,\nfor example, type checking, memory allocation and garbage collection\nthat can slow down the execution are carried out by the transpiled\nprogram too.\n\n*Cython* is a programming language whose goal is to\nfacilitate writing C extensions for the Python language. In\nparticular, it can translate (optionally) statically typed Python-like\ncode into C code that relies on *CPython*. Similarly to the\nconsiderations made for *Nuitka*, this makes it a powerful tool\nwhenever it is possible to rely on *libpython* (and when its\noverhead is negligible, i.e., when dealing with sufficiently large\nscale computations), but not in the context of interest here.\n\nFinally, although it does not use Python as source language, we should\nmention that *Julia* too is just-in-time (and partially\nahead-of-time) compiled into LLVM\ncode. The emitted LLVM code relies however on the *Julia*\nruntime library such that considerations similar to the one made for\n*Cython* and *Nuitka* apply.\n\n### *prometeo*'s transpiler\nTranspilation of programs written using a restricted subset of the\nPython language into C programs is carried out using *prometeo*'s\ntranspiler. This source-to-source transformation tool analyzes abstract\nsyntax trees (AST) associated with the source files to be transpiled\nin order to emit high-performance and embeddable C code. In order to\ndo so, special rules need to be imposed on the Python code. This makes\nthe otherwise extremely challenging task of transpiling an interpreted high-level\nduck-typed language into a compiled low-level statically typed one\npossible. In doing so, we define what is sometimes referred to as an *embedded* \nDSL in the sense the resulting language uses the syntax of a host language\n(Python itself) and, in *prometeo*'s case, it can also be executed by\nthe standard Python interpreter. \n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/zanellia/prometeo/blob/master/figures/simple_ast_annotated.png\" width=\"70%\" \u003e\n\u003c/p\u003e\n\n### a more advanced example (Riccati factorization)\n```python\nfrom prometeo import *\n\nnx:  dims = 2\nnu:  dims = 2\nnxu: dims = nx + nu\nN:   dims = 5\n\ndef main() -\u003e int:\n\n    # number of repetitions for timing\n    nrep : int = 10000\n\n    A: pmat = pmat(nx, nx)\n    A[0,0] = 0.8\n    A[0,1] = 0.1\n    A[1,0] = 0.3\n    A[1,1] = 0.8\n\n    B: pmat = pmat(nx, nu)\n    B[0,0] = 1.0  \n    B[1,1] = 1.0\n\n    Q: pmat = pmat(nx, nx)\n    Q[0,0] = 1.0  \n    Q[1,1] = 1.0\n\n    R: pmat = pmat(nu, nu)\n    R[0,0] = 1.0  \n    R[1,1] = 1.0\n\n    A: pmat = pmat(nx, nx)\n    B: pmat = pmat(nx, nu)\n    Q: pmat = pmat(nx, nx)\n    R: pmat = pmat(nu, nu)\n\n    RSQ: pmat = pmat(nxu, nxu)\n    Lxx: pmat = pmat(nx, nx)\n    M: pmat = pmat(nxu, nxu)\n    w_nxu_nx: pmat = pmat(nxu, nx)\n    BAt : pmat = pmat(nxu, nx)\n    BA : pmat = pmat(nx, nxu)\n    pmat_hcat(B, A, BA)\n    pmat_tran(BA, BAt)\n\n    RSQ[0:nu,0:nu] = R\n    RSQ[nu:nu+nx,nu:nu+nx] = Q\n\n    # array-type Riccati factorization\n    for i in range(nrep):\n        pmt_potrf(Q, Lxx)\n        M[nu:nu+nx,nu:nu+nx] = Lxx\n        for i in range(1, N):\n            pmt_trmm_rlnn(Lxx, BAt, w_nxu_nx)\n            pmt_syrk_ln(w_nxu_nx, w_nxu_nx, RSQ, M)\n            pmt_potrf(M, M)\n            Lxx[0:nx,0:nx] = M[nu:nu+nx,nu:nu+nx]\n\n    return 0\n\n```\nSimilarly, the code above ([`example/riccati/riccati_array.py`](https://github.com/zanellia/prometeo/blob/master/examples/riccati_example/riccati_array.py)) can be run by the standard Python interpreter using the command `pmt riccati_array.py --cgen=False` and prometeo can generate, compile and run C code using instead `pmt riccati_array.py --cgen=True`.\n\n### supported Python constructs\nIn order to be able to transpile to C, only a subset of the Python language is supported. However, non C-like features such as function overload and classes are supported by prometeo's transpiler. The adapted Riccati example ([`examples/riccati/riccati_mass_spring_2.py`](https://github.com/zanellia/prometeo/blob/master/examples/riccati_example/riccati_mass_spring_2.py)) below shows how classes can be created and used.\n\n```python\nfrom prometeo import *\n\nnm: dims = 4\nnx: dims  = 2*nm\nsizes: dimv = [[8,8], [8,8], [8,8], [8,8], [8,8]]\nnu: dims  = nm\nnxu: dims = nx + nu\nN:  dims  = 5\n\nclass qp_data:\n    A: List = plist(pmat, sizes)\n    B: List = plist(pmat, sizes)\n    Q: List = plist(pmat, sizes)\n    R: List = plist(pmat, sizes)\n    P: List = plist(pmat, sizes)\n\n    fact: List = plist(pmat, sizes)\n\n    def factorize(self) -\u003e None:\n        M: pmat = pmat(nxu, nxu)\n        Mxx: pmat = pmat(nx, nx)\n        L: pmat = pmat(nxu, nxu)\n        Q: pmat = pmat(nx, nx)\n        R: pmat = pmat(nu, nu)\n        BA: pmat = pmat(nx, nxu)\n        BAtP: pmat = pmat(nxu, nx)\n        pmat_copy(self.Q[N-1], self.P[N-1])\n\n        pmat_hcat(self.B[N-1], self.A[N-1], BA)\n        pmat_copy(self.Q[N-1], Q)\n        pmat_copy(self.R[N-1], R)\n        for i in range(1, N):\n            pmat_fill(BAtP, 0.0)\n            pmt_gemm_tn(BA, self.P[N-i], BAtP, BAtP)\n\n            pmat_fill(M, 0.0)\n            M[0:nu,0:nu] = R\n            M[nu:nu+nx,nu:nu+nx] = Q\n\n            pmt_gemm_nn(BAtP, BA, M, M)\n            pmat_fill(L, 0.0)\n            pmt_potrf(M, L)\n\n            Mxx[0:nx, 0:nx] = L[nu:nu+nx, nu:nu+nx]\n\n            # pmat_fill(self.P[N-i-1], 0.0)\n            pmt_gemm_nt(Mxx, Mxx, self.P[N-i-1], self.P[N-i-1])\n            # pmat_print(self.P[N-i-1])\n\n        return\n\ndef main() -\u003e int:\n\n    A: pmat = pmat(nx, nx)\n    Ac11 : pmat = pmat(nm,nm)\n    Ac12 : pmat = pmat(nm,nm)\n    for i in range(nm):\n        Ac12[i,i] = 1.0\n\n    Ac21 : pmat = pmat(nm,nm)\n    for i in range(nm):\n        Ac21[i,i] = -2.0\n\n    for i in range(nm-1):\n        Ac21[i+1,i] = 1.0\n        Ac21[i,i+1] = 1.0\n\n    Ac22 : pmat = pmat(nm,nm)\n\n    for i in range(nm):\n        for j in range(nm):\n            A[i,j] = Ac11[i,j]\n\n    for i in range(nm):\n        for j in range(nm):\n            A[i,nm+j] = Ac12[i,j]\n\n    for i in range(nm):\n        for j in range(nm):\n            A[nm+i,j] = Ac21[i,j]\n\n    for i in range(nm):\n        for j in range(nm):\n            A[nm+i,nm+j] = Ac22[i,j]\n\n    tmp : float = 0.0\n    for i in range(nx):\n        tmp = A[i,i]\n        tmp = tmp + 1.0\n        A[i,i] = tmp\n\n    B: pmat = pmat(nx, nu)\n\n    for i in range(nu):\n        B[nm+i,i] = 1.0\n\n    Q: pmat = pmat(nx, nx)\n    for i in range(nx):\n        Q[i,i] = 1.0\n\n    R: pmat = pmat(nu, nu)\n    for i in range(nu):\n        R[i,i] = 1.0\n\n    qp : qp_data = qp_data() \n\n    for i in range(N):\n        qp.A[i] = A\n\n    for i in range(N):\n        qp.B[i] = B\n\n    for i in range(N):\n        qp.Q[i] = Q\n\n    for i in range(N):\n        qp.R[i] = R\n\n    qp.factorize()\n    \n    return 0\n```\n\n__Disclaimer: prometeo is still at a very preliminary stage and only a few linear algebra operations and Python constructs are supported for the time being.__\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzanellia%2Fprometeo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzanellia%2Fprometeo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzanellia%2Fprometeo/lists"}