{"id":26059624,"url":"https://github.com/mnikander/compiler_fragments","last_synced_at":"2025-03-08T13:26:46.214Z","repository":{"id":275792238,"uuid":"923059050","full_name":"mnikander/compiler_fragments","owner":"mnikander","description":"Transpile symbolic expressions to C++","archived":false,"fork":false,"pushed_at":"2025-03-04T09:28:35.000Z","size":100,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-04T09:34:47.639Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mnikander.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-27T15:19:30.000Z","updated_at":"2025-03-04T09:28:38.000Z","dependencies_parsed_at":"2025-02-11T22:23:39.042Z","dependency_job_id":null,"html_url":"https://github.com/mnikander/compiler_fragments","commit_stats":null,"previous_names":["mnikander/compiler_fragments"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mnikander%2Fcompiler_fragments","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mnikander%2Fcompiler_fragments/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mnikander%2Fcompiler_fragments/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mnikander%2Fcompiler_fragments/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mnikander","download_url":"https://codeload.github.com/mnikander/compiler_fragments/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242555271,"owners_count":20148665,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-08T13:26:45.475Z","updated_at":"2025-03-08T13:26:46.195Z","avatar_url":"https://github.com/mnikander.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Compiler fragments\n\nThis repository contains code fragments for the _intermediate code generation_ stage of a compiler, for a Lisp-like language.\nThe aim is to experiment with a few language features and learn more about compilers.\nThe following [symbolic expression](https://en.wikipedia.org/wiki/S-expression):\n\n```lisp\n(define x 5)\n```\ncan easily be parsed into the following abstract syntax tree, in JSON form:\n\n```json\n[\"define\", \"x\", 5]\n```\n\nThis AST can then be transpiled into C++ code:\n\n```c++\nconst auto x = 5;\n```\n\n## Getting Started\n\n1. Clone this repo\n2. Ensure you have _nodejs_, _npm_, and _g++_ installed\n3. `npm run main` to build and run the example\n4. `npm test` to build and run the unit tests\n\nBoth the main function and the unit tests will automatically transpile their abstract syntax trees to C++ and then compile and execute the resulting programs. \nIf you wish to build the main program without immediately executing it, you can use `npm run build` and then manually execute it with `./out/artifacts/main`.\nYou can find all generated source files, executables, and result text files, in the directory `out/artifacts`.\n\n## System Design\nThe following sections outline key design decisions, the system design, as well as the pipeline employed for development and testing.\n\n### Choice of the Implementation Language\n\nThe code generator itself is written in TypeScript, in the hopes of running it inside of a browser one day.\n_NodeJS_ and _npm_ also provide easy access to a huge number of libraries and modules.\nCompared to a language such as C++, this can speed up development significantly.\nThe runtime performance of the _code generator_ is currently not a concern, since the goal is to prototype a few language features and not to create a production grade compiler.\nThis code generator could also be integrated into a project built with [Langium](https://langium.org/) to get syntax highlighting and auto-completion in VS code.\n\n### Choice of the Target Language\n\nSeveral target languages were considered: LLVM IR, WebAssembly, C, C++, JavaScript, and TypeScript.\nThe web-based languages allow executing the program in a browser, which is great for usability.\nThe assembly languages could provide the best runtime performance.\nC++ was chosen as the target language, however, because it's fast and provides useful abstractions.\nThis makes it easier to develop and debug the code generator, than if LLVM IR or WebAssembly were used instead.\nIt is highly beneficial, though, if the architecture and testing pipelines support adding another target language later on.\n\n### Notation for Abstract Syntax Trees\n\nDepending on how the grammar is defined, the AST can become very complicated, very quickly.\nThe expression `(display (add 1 (add 2 4)))` _could_ be parsed to a huge AST such as:\n\n```json\n{\n    \"$type\": \"Display\",\n    \"value\": {\n        \"$type\": \"Application\",\n        \"value\": [\n            {\n                \"$type\": \"Abstraction\",\n                \"value\": \"add\"\n            },\n            {\n                \"$type\": \"Integer\",\n                \"value\": 1\n            },\n            {\n                \"$type\": \"Application\",\n                \"value\": [\n                    {\n                        \"$type\": \"Abstraction\",\n                        \"value\": \"add\"\n                    },\n                    {\n                        \"$type\": \"Integer\",\n                        \"value\": 2\n                    },\n                    {\n                        \"$type\": \"Integer\",\n                        \"value\": 4\n                    }\n                ]\n            }\n        ]\n    }\n}\n```\n\nLisp-like languages have a [famously simple grammar](https://iamwilhelm.github.io/bnf-examples/lisp), though.\nThe explicit parentheses of the symbolic expressions make it very easy to create an AST in JSON form:\n```json\n{\n    \"symbolic_expression\": \"(display (add 1 (add 2 4)))\",\n    \"json_ast\" :  [\"display\", [\"add\", 1, [\"add\", 2, 4]]]\n}\n```\nThis far more compact notation is much easier to write and work with than the long-form notation above.\nFor these reasons, this minimalist approach was chosen.\n\nAdding type-annotations would make the AST more complicated.\nEvery element in the JSON array must be replaced by an object.\nThis representation would still be quite simple, though:\n\n```json\n{\n    \"symbolic_expression\": \"(display (add 1 (add 2 4)))\",\n    \"json_ast\":\n    [\n        {\"display\": \"i64 -\u003e Output\"},\n        [\n            {\"add\": \"[i64 i64] -\u003e i64\"},\n            {\"1\": \"i64\"},\n            [\n                {\"add\": \"[i64 i64] -\u003e i64\"},\n                {\"2\": \"i64\"},\n                {\"4\": \"i64\"}\n            ]\n        ]\n    ]\n}\n```\n\n### Test Design\n\nTesting the code generator effectively, presents a number of challenges:\n1. How can the correctness of the code generation be ensured?\n2. How can unit tests for code generation be written easily?\n3. How can the maintenance effort for unit tests be kept low, even if the code generation is frequently refactored?\n4. How could the code generation for two different target languages be tested effectively?\n5. How can alternative implementations for the same language feature be tested?\n\nIn general, there are several options for how to test the code generation including:\n1. string comparison on the generated C++ code\n2. snapshot testing, where the generated code is compared to an earlier snapshot of the generated code\n3. compile and execute the generated code\n\n#### 1. String Comparison\nThe string comparison approach executes quickly and ensures that the code is exactly what is expected.\nThere are many ways to generate C++ code for one particular AST node, however.\nFor example: `const int x = 5;` and `int const x = 5;` mean the same thing.\nAn expression such as `(lambda (a b) a)` can be translated into C++ lambda expressions or into function objects.\nHaving to modify the unit tests, every time the code generation is tweaked slightly, could be very time-consuming.\nFurthermore, test cases via string comparison would have to be re-implemented from scratch for each additional target language.\nThis makes testing via string comparisons unattractive for this project.\n\n#### 2. Snapshot testing\nSnapshot testing is easy to implement: just add the generated C++ files to the git repository and keep an eye out for changes.\nSnapshot testing doesn't check the correctness of the code though.\nIt will probably be used, but not as the primary test mechanism.\n\n#### 3. Compilation and Execution\nThe last option, to compile and execute the generated code can be tedious to implement.\nIt can also take a while to run a large number of test-cases.\nThis can be mitigated by executing only those test cases which are effected by recently modified files.\nThe chosen unit testing framework, Vitest, does exactly this.\n\nThis overall testing strategy means that the result of the executed code must be passed to the unit test.\nThis can be done by printing to stdout or writing the result to a file.\nA test for addition could be `(display (+ 1 2))`, i.e. the expression to be tested is wrapped inside the command to print to the console.\nSince these tests only check the output of the executed program, they are agnostic to implementation details.\nOther than the call to the compilation pipeline, these tests are also agnostic to the target language.\nThis should keep the amount of maintenance work low, as the codebase evolves.\n\nTesting only the output of the executable does have some drawbacks, however.\nSome properties of the generated code cannot be tested directly.\nThis may require extra test cases.\nFortunately, most black-box tests should be fast and easy to write.\n\nThe C++ toolchain for unit testing has the following structure:\n\n```\n+-----------------------+\n| TypeScript unit test  |\n| - JSON string for AST |\n| - Expected result     |\n+-----------------------+\n           v\n           v  ... generate code\n           v\n  +------------------+\n  | C++ source files |\n  +------------------+\n           v\n           v  ... call a C++ compiler\n           v\n    +-------------+\n    | Executable  |\n    +-------------+\n           v\n           v  ... run executable and\n           v      pipe output to a file\n           v\n    +-------------+\n    | Result file |\n    +-------------+\n           v\n           v\n           v\n  +------------------+\n  | Check output vs. |\n  | expected value   |\n  +------------------+\n```\n\n**Note**: Piping the output to a file is optional, since stdout can be read by the unit test directly.\nIt _can_ be helpful for debugging purposes, but this step might be removed in the future, for simplicity.\n\n---\n**Copyright (c) 2025 Marco Nikander**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmnikander%2Fcompiler_fragments","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmnikander%2Fcompiler_fragments","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmnikander%2Fcompiler_fragments/lists"}