{"id":16909634,"url":"https://github.com/yjdoc2/configurable-custom-keyword-lexer","last_synced_at":"2025-03-21T00:40:20.401Z","repository":{"id":100666466,"uuid":"406803757","full_name":"YJDoc2/Configurable-Custom-Keyword-Lexer","owner":"YJDoc2","description":"This shows proof-of-concept implementation of lexer-parser-evaluator which allows setting custom values to keywords.","archived":false,"fork":false,"pushed_at":"2021-09-17T05:22:07.000Z","size":76,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-25T21:12:22.860Z","etag":null,"topics":["language","lexer","parser"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/YJDoc2.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"License","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-15T14:35:49.000Z","updated_at":"2021-09-18T03:35:29.000Z","dependencies_parsed_at":"2023-05-16T15:45:21.980Z","dependency_job_id":null,"html_url":"https://github.com/YJDoc2/Configurable-Custom-Keyword-Lexer","commit_stats":{"total_commits":11,"total_committers":1,"mean_commits":11.0,"dds":0.0,"last_synced_commit":"a350d3243d0770b015a2cd53dd6059833f88e9ed"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YJDoc2%2FConfigurable-Custom-Keyword-Lexer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YJDoc2%2FConfigurable-Custom-Keyword-Lexer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YJDoc2%2FConfigurable-Custom-Keyword-Lexer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YJDoc2%2FConfigurable-Custom-Keyword-Lexer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/YJDoc2","download_url":"https://codeload.github.com/YJDoc2/Configurable-Custom-Keyword-Lexer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244717365,"owners_count":20498284,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["language","lexer","parser"],"created_at":"2024-10-13T18:56:41.678Z","updated_at":"2025-03-21T00:40:20.378Z","avatar_url":"https://github.com/YJDoc2.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Custom Configurable Lexer-Parser\n\n---\n\n### Note\n\nThis is still very experimental, and for any syntax error it will just panic giving very unhelpful error message. The error recovery and more helpful error messages are possible, but have not been implemented yet. Also this is more of a proof-of-concept, and the language itself only supports variable declaration,assignment, while and for loop, if-else statements and baked-in print statement\n\nIf someone is actually interested in this, I can add more examples, write up a better documentation etc, let me know in issues.\nThis can also theoretically be used to convert program from one keyword mappings to another, but haven't gotten around to do that yet.\n\n---\n\nInspired by [a tweet](https://twitter.com/Felienne/status/1435864809963130883) , I made this lexer-parser pair along with an evaluator, which allows you to set custom keywords for the language using a config file, and according to that config, this runs the program. To show in short, what this makes possible:\n\n```sh\n# English\nlet v1 = 1234;\nif (v1\u003e12){\n    print(v1+(123 - (25*6)));\n} else {\n    print('Hello');\n}\nv1 = 1;\nif(v1\u003e12){\n    print(v1+(123 - (25*6)));\n} else{\n    print('Hello');\n}\nlet i = 0;\nwhile (i\u003c5){\n    print('आता i ची value आहे '+i);\n    i = i+1;\n}\nfor k in [i,v1]{\n    print(k);\n}\n\n```\n\nAnd\n\n```sh\n# Marathi\nनवीन v1 = १२३४;\nजर (v1\u003e12) तर{\n    हे(v1+(123 - (25*6))) दाखवा ;\n} नाहीतर {\n    हे('Hello') दाखवा;\n}\nv1 = १;\nजर (v1\u003e12) तर{\n    हे(v1+(123 - (25*6))) दाखवा;\n} नाहीतर {\n    हे('Hello') दाखवा;\n}\nनवीन i = 0;\nजोपर्यंत (i\u003c5) तोपर्यंत{\n    हे('आता i ची value आहे '+i)दाखवा;\n    i = i+1;\n}\nfor k in [i,v1]{\n    हे(k)दाखवा;\n}\n```\n\nboth are a valid programs and can be run by \u003cstrong\u003ethe same binary\u003c/strong\u003e, with their own config-english and config-marathi config files. Moreover, you can create custom config files to write code in your own language.\n\n## How this works\n\nThis is made of three parts : A handwritten lexer, taken much after the amazing [Crafting Interpreters](https://craftinginterpreters.com/), A Parser , generated using [Lalrpop crate](https://crates.io/crates/lalrpop), and an evaluator.\n\nthe commandline interface is :\n\n```\nUSAGE:\n    config_lex --config \u003cCONFIG\u003e --file \u003cFILE\u003e\n\nFLAGS:\n    -h, --help       Print help information\n    -V, --version    Print version information\n\nOPTIONS:\n    -c, --config \u003cCONFIG\u003e    Keyword Configuration file\n    -f, --file \u003cFILE\u003e        Source code file\n```\n\nThis takes in a keyword config file, which specify the keyword mappings, which are then used by the lexer to determine the tokens. The lexer then emits the tokens, which are used by parser to generate AST. This AST is evaluated using a recursive tree walker, again taken after [Crafting Interpreters](https://craftinginterpreters.com/).\n\nThe point to note here is that the major changes here from other lexer-parsers are confined to lexer and parser only. This can still emit a convenient Intermediate Representation, such as AST, which can then be transplied to some other language.\n\n### Language Specification\n\nCurrently following keywords are configurable, the brackets are default english fallback values:\n\n- PrintStart ( print )\n- PrintEnd\n- ForStart ( for )\n- ForAux1\n- ForAux2\n- In ( in )\n- ForAux3\n- ForAux4\n- IfStart ( if )\n- IfAux1\n- IfAux2\n- ElseStart ( else )\n- ElseAux1\n- LetStart ( let )\n- Or ( || )\n- And ( \u0026\u0026 )\n- WhileStart ( while )\n- WhileAux1\n- WhileAux2\n\nWhere \\*Aux keywords are optional, and other are required. These auxillary keywords are provided so that the constructs, such as : if, while can be made more \"organic\", as some languages can use extra keywords to make the constructs more \"coherent\"/\"natural\" for that language.\n\nCurrently the structure of this language is as (can be seen in parser.lalrpop file):\n\n```sh\n-\u003e This supports only string and numerical datatypes, and is dynamically typed\n-\u003e expr is any expression containing +-/*() and literal values or variables.\n-\u003e condition is value/variables compared using \u003c,\u003e,\u003c=,=\u003e,==,!=\n    and such conditions joined by And,Or tokens.\n-\u003e While comparing string, it only == and != do a char-by-char comparison,\n    others compare by string length\n-\u003e strings can be only added with other strings or numbers (like java)\n    and other arithmetic operations are invalid for strings.\n-\u003e block is statements inside { and }\n\nprint =\u003e PrintStart ( Expr ) PrintEnd\nlet =\u003e LetStart ID = Expr\nif =\u003e IfStart IfAux1 (condition) IfAux2 block ElseStart ElseAux1 block # else section is optional\nwhile =\u003e WhileStart WhileAux1 (condition) WhileAux2 block\nfor =\u003e ForStart ForAux1 ID ForAux2 ForAux3 In array ForAux4 block\narray =\u003e [ comma-separated-expr ]\n\n```\n\n---\n\n## Licence\n\nThis code is released under GNU GPL V3, see [License](https://github.com/YJDoc2/Configurable-Custom-Keyword-Lexer/blob/main/License) file for more info.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyjdoc2%2Fconfigurable-custom-keyword-lexer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyjdoc2%2Fconfigurable-custom-keyword-lexer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyjdoc2%2Fconfigurable-custom-keyword-lexer/lists"}