{"id":13343208,"url":"https://github.com/jnoortheen/parser-experiments","last_synced_at":"2025-03-12T04:32:48.811Z","repository":{"id":86397762,"uuid":"490649117","full_name":"jnoortheen/parser-experiments","owner":"jnoortheen","description":"Python parsing experiments","archived":true,"fork":false,"pushed_at":"2023-04-07T09:39:06.000Z","size":1102,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-10-24T15:38:47.352Z","etag":null,"topics":["xonsh","xonsh-dev"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jnoortheen.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-05-10T10:30:56.000Z","updated_at":"2024-06-18T16:34:02.000Z","dependencies_parsed_at":"2024-10-24T06:13:45.678Z","dependency_job_id":"c307bc01-6f9c-4fc7-b119-68097bcf66e3","html_url":"https://github.com/jnoortheen/parser-experiments","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jnoortheen%2Fparser-experiments","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jnoortheen%2Fparser-experiments/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jnoortheen%2Fparser-experiments/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jnoortheen%2Fparser-experiments/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jnoortheen","download_url":"https://codeload.github.com/jnoortheen/parser-experiments/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243158974,"owners_count":20245668,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["xonsh","xonsh-dev"],"created_at":"2024-07-29T19:30:42.612Z","updated_at":"2025-03-12T04:32:48.800Z","avatar_url":"https://github.com/jnoortheen.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Overview\n\nexperiment with different ways to parse and generate ast with optimal speed \u0026 memory. \nAll of these below parse Python3.10 grammar. \n\n| module                   | total allocated size | time  | peak       |\n|--------------------------|----------------------|-------|------------|\n| pgen2                    | 369KiB               | 0.08s | 684KiB     |\n| pegen + regex(tokenizer) | 1767KiB              | 0.32s | 2015KiB    |\n| pegen                    | 1234KiB              | 0.38s | 2281.9KiB  |\n| xonsh-ply                | 8240.6 KiB           | 0.65s | 10333.5KiB |\n| lark (lalr-cached)       | 3753.7 KiB           | 0.74s | 9307.3KiB  |\n| parso                    | 3542.7 KiB           | 0.80s | 3690.2KiB  |\n| treesitter               | 9137.0 KiB           | 1.56s | 9708.7KiB  |\n| libcst                   | 21817.KiB            | 6.5s  | 23024.4KiB |\n\n\nseems like both are good. easpecially pgen2 interms of memory usage and performance. but we can use pegen2 \nas it has a separate pypi package. We can expect some stability as Python may include more and more peg only changes\n\n# Conclusions\n\n## A. [xonsh-ply](https://www.dabeaz.com/ply/ply.html#ply_nn11)\n\n- the existing parser is slow and uses more memory. \n- the ply codebase is a mess though rply is good and we can optimize with some care\n\n## B. [pegen](https://github.com/we-like-parsers/pegen)\n\n- It will be following the official parser, hence future proof\n- generates AST which we can feed directly to the interpreter\n- has big peak memory size but it gets released and will end up with optimal size\n  - when regex is used to tokenize the peak memory is `2015KiB`\n- I found a PR which intends to make use of pegen in place of ply\n  - https://github.com/nucleic/enaml/pull/474/\n  - https://github.com/jecki/DHParser - another interesting peg generator. \n    - includes test generation \n    - language server...\n\n## C. pgen2\n\n- it comes from lib2to3 package of CPython. but it will be removed in py3.13 or so ... not much future proof\n  - but black-formatter has forked it and it may stick around sometime more. \n  we can refer these packages if we decided to base our parser on this\n- but has very less memory usage and faster too for any of the tested tools here\n- Links\n  - https://github.com/pyga/awpa/tree/master/awpa\n  - parso grammars\n\n## D. parso\n \n- it is a fork of pgen2\n- does error recovery of sorts and hence the high memory usage\n- we can pick some pieces from this project if we decided to use pgen2\n\n## E. treesitter\n\n- even with the python bindings it ended up using more memory. \n- seems like the memory is not freed as the peak memory is the same as total allocated.\n\n# Step forward\n\n1. implement the completion-context parser in `pgen2` and `pegen` and compare the\n   1. development time\n   2. performance\n   3. memory usage\n\n\n# Links\n\n- https://www.dabeaz.com/ply/ply.html - good resource to dive into the world of parser generation\n- https://itnext.io/hacking-the-python-syntax-part-1-ternary-operator-bbcb04aa6ecb\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjnoortheen%2Fparser-experiments","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjnoortheen%2Fparser-experiments","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjnoortheen%2Fparser-experiments/lists"}