{"id":17442571,"url":"https://github.com/jez/smlnj-viscomp-example","last_synced_at":"2025-12-12T13:30:14.259Z","repository":{"id":43553905,"uuid":"511414589","full_name":"jez/smlnj-viscomp-example","owner":"jez","description":"An example of how to use SML/NJ's Visible Compiler APIs","archived":false,"fork":false,"pushed_at":"2022-07-07T06:58:43.000Z","size":20,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-08T18:17:46.796Z","etag":null,"topics":["sml","smlnj"],"latest_commit_sha":null,"homepage":"","language":"Standard ML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jez.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-07-07T06:46:28.000Z","updated_at":"2022-10-14T17:49:14.000Z","dependencies_parsed_at":"2022-07-08T00:09:10.549Z","dependency_job_id":null,"html_url":"https://github.com/jez/smlnj-viscomp-example","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jez%2Fsmlnj-viscomp-example","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jez%2Fsmlnj-viscomp-example/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jez%2Fsmlnj-viscomp-example/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jez%2Fsmlnj-viscomp-example/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jez","download_url":"https://codeload.github.com/jez/smlnj-viscomp-example/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239679608,"owners_count":19679500,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["sml","smlnj"],"created_at":"2024-10-17T16:17:55.681Z","updated_at":"2025-12-12T13:30:14.220Z","avatar_url":"https://github.com/jez.png","language":"Standard ML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# smlnj-viscomp-example\n\nAn example of how to use [SML/NJ]'s woefully under-documented [Visible Compiler]\nto build something that could possibly be used to build a language server.\n\nIt's not finished and I have no plans to do so. Instead, it's meant to hint at\nan implementation or architecture that might interest you, as well as showcase\nAPIs that (as of SML/NJ v110.99.2) were known to compile. As in: it mostly just\nserves to jog my memory of what *can* be done, not be an example of how best to\nuse these APIs.\n\n## Demo\n\nIf you have SML/NJ v110.99.2 installed, you can run this to build the project,\n(using my [symbol] build tool, which is just a shell script + Makefile that's\nchecked into the repo):\n\n```\n❯ ./symbol make\n[ .. ] Analyzing CM dependencies...\n[ .. ] Building heap image with SML/NJ...\n[ .. ] Building 'smlnj-viscomp-example' into '.symbol-work/bin'...\n[ OK ] .symbol-work/bin/smlnj-viscomp-example\n```\n\nOnce that's done, here are some ways you can interact with it from the command\nline:\n\n```\n❯ .symbol-work/bin/smlnj-viscomp-example --help\nShows the type of the expression variable at the location\nUsage:\n  .symbol-work/bin/smlnj-viscomp-example [options] \u003cfilename\u003e \u003ccharpos\u003e\n\nArguments:\n  \u003cfilename\u003e      The single SML file to run on.\n  \u003ccharpos\u003e       The 1-based character offset in the file to query.\n\nOptions:\n  -h, --help      Show this usage message.\n\nExample:\n  .symbol-work/bin/smlnj-viscomp-example foo.sml 12\n```\n\n```\n❯ cat example.sml\nfun fakePrint msg = ()\nval greeting = \"Hello, world!\"\nval _ = fakePrint greeting\n\n❯ .symbol-work/bin/smlnj-viscomp-example example.sml 30\nFirst query result:\nval greeting = \"Hello, world!\"\n\nType:\n?.string\n\n❯ .symbol-work/bin/smlnj-viscomp-example example.sml 66\nFirst query result:\nfakePrint\n\nType:\n'a -\u003e ?.unit\n```\n\n## Explanation\n\nThe SML/NJ wraps certain AST nodes in `MARK...` nodes, like `MARKdec`,\n`MARKexp`, etc. These nodes contain the nested AST node and also a `region`,\nwhich is a begin and end offset corresponding to locations in the source buffer.\n\nGiven a character offset, we can walk the AST that SML/NJ produces and\naccumulate a list of matching AST nodes where the cursor position lies within\nthe containing region. If we combine this with a pre-order traversal of the\ntree and push results onto the front of a list as we go, the resulting list will\nbe ordered from most specific results to least specific results, so we can take\nthe head of the list and find the type of it.\n\nThe traversal is implemented (`Query.atPos`) but getting the type of an\narbitrary result is mostly not, except for the case of a variable in a `val`\nbinding, as a proof of concept.\n\nThe idea would be that these two APIs (location-based queries and query results\nwrapping typed AST nodes) could be used to power IDE features like \"show me the\ntype\" or \"go to definition\"\n\n## Gotchas / future work\n\n- This only works for a single file. To do multiple files, it would require\n  figuring out the APIs to use the Compilation Manager (CM) structures to load\n  and type check a file in the context of a number of files and libraries.\n\n- It currently prints out `?.` in front of all the types...\n\n  Honestly, I think this has to do with the next gotcha, which is:\n\n- It doesn't even typecheck a file in the context of the pervasive/basis\n  library. Again, it's likely that integrating with CM would fix this?\n\nThe APIs **should** all be there, it's just not documented. When I was looking\ninto this last, I basically grepped for `elabTop` in the SML/NJ sources, found\none result in `base/compiler/TopLevel/main/compile.sml`, and was in the process\nof working my way out from there to see how it sets up the environment that gets\nfed into `elabTop`. Presumably all we have to do is mimic enough of that code\nthat it would work (or maybe there's an even higher level API we could call that\nboth takes care of setting up the right environment and gives us back an `Absyn`\ntyped AST).\n\n## Understanding the Visible Compiler APIs\n\nThe online docs for the SML/NJ Visible Compiler APIs are terrible.\n\nYou're going to have much better luck searching through and reading the SML/NJ\nsource. It's important that you look at the source for the right version,\nbecause these APIs tend to break in minor ways from version to version.\n\nSome ways to get the sources for the version relevant to you:\n\n-   If you know how to use SVN (or want to learn):\n\n    ```\n    svn checkout --username anonsvn --password anonsvn https://smlnj-gforge.cs.uchicago.edu/svn/smlnj/\n    ```\n\n    and then figure out how to checkout the version you're using\n\n-   Otherwise, use the \"install from source\" instructions on the release page for\n    your version of SML/NJ, for example:\n\n    \u003chttps://www.smlnj.org/dist/working/110.99.2/index.html\u003e\n\n    And be sure to follow the suggestion to edit `config/targets` so that you\n    uncomment whatever is required to get the compiler source code downloaded\n    too. For 110.99.2, this amounted to ensuring that this directive was\n    uncommented in the `config/targets` file:\n\n    ```\n    # unpack the source code for everything (including for the SML/NJ compiler\n    # itself); this is not required,  unless you are doing compiler hacking,\n    # but it may be interesting to look at.\n    #\n    request src-smlnj\n    ```\n\n-   If you're really lazy, you can probably get by with using the SVN web viewer\n\n    \u003chttps://smlnj-gforge.cs.uchicago.edu/scm/viewvc.php/?root=smlnj\u003e\n\n    but it takes some getting used to, and you're very quickly want to be able\n    to start grepping, so you probably only want to use this in a pinch.\n\nOnce you've got the sources, the `base/compiler` folder is the one where all the\ngood stuff is. If you chose an SVN option above (instead of downloading just the\nsources for the given version), you'll have to make sure that you're in the\nright `base/compiler` folder. For 110.99.2, that folder is\n`/sml/releases/release-110.99.2/base/compiler`.\n\nOnce you have the SML/NJ sources, you should be able to work from the\ndependencies in this example project's CM file to the relevant files in your\nSML/NJ checkout to begin browsing what APIs are available to you.\n\n\n[SML/NJ]: \u003chttps://www.smlnj.org/\u003e\n[Visible Compiler]: \u003chttps://www.smlnj.org/doc/Compiler/pages/compiler.html\u003e\n[symbol]: https://github.com/jez/symbol\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjez%2Fsmlnj-viscomp-example","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjez%2Fsmlnj-viscomp-example","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjez%2Fsmlnj-viscomp-example/lists"}