Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/jez/smlnj-viscomp-example

An example of how to use SML/NJ's Visible Compiler APIs
https://github.com/jez/smlnj-viscomp-example

sml smlnj

Last synced: 8 months ago
JSON representation

An example of how to use SML/NJ's Visible Compiler APIs

Awesome Lists containing this project

README 

          
# smlnj-viscomp-example



An example of how to use [SML/NJ]'s woefully under-documented [Visible Compiler]

to build something that could possibly be used to build a language server.



It's not finished and I have no plans to do so. Instead, it's meant to hint at

an implementation or architecture that might interest you, as well as showcase

APIs that (as of SML/NJ v110.99.2) were known to compile. As in: it mostly just

serves to jog my memory of what *can* be done, not be an example of how best to

use these APIs.



## Demo



If you have SML/NJ v110.99.2 installed, you can run this to build the project,

(using my [symbol] build tool, which is just a shell script + Makefile that's

checked into the repo):



```

❯ ./symbol make

[ .. ] Analyzing CM dependencies...

[ .. ] Building heap image with SML/NJ...

[ .. ] Building 'smlnj-viscomp-example' into '.symbol-work/bin'...

[ OK ] .symbol-work/bin/smlnj-viscomp-example

```



Once that's done, here are some ways you can interact with it from the command

line:



```

❯ .symbol-work/bin/smlnj-viscomp-example --help

Shows the type of the expression variable at the location

Usage:

  .symbol-work/bin/smlnj-viscomp-example [options]  



Arguments:

        The single SML file to run on.

         The 1-based character offset in the file to query.



Options:

  -h, --help      Show this usage message.



Example:

  .symbol-work/bin/smlnj-viscomp-example foo.sml 12

```



```

❯ cat example.sml

fun fakePrint msg = ()

val greeting = "Hello, world!"

val _ = fakePrint greeting



❯ .symbol-work/bin/smlnj-viscomp-example example.sml 30

First query result:

val greeting = "Hello, world!"



Type:

?.string



❯ .symbol-work/bin/smlnj-viscomp-example example.sml 66

First query result:

fakePrint



Type:

'a -> ?.unit

```



## Explanation



The SML/NJ wraps certain AST nodes in `MARK...` nodes, like `MARKdec`,

`MARKexp`, etc. These nodes contain the nested AST node and also a `region`,

which is a begin and end offset corresponding to locations in the source buffer.



Given a character offset, we can walk the AST that SML/NJ produces and

accumulate a list of matching AST nodes where the cursor position lies within

the containing region. If we combine this with a pre-order traversal of the

tree and push results onto the front of a list as we go, the resulting list will

be ordered from most specific results to least specific results, so we can take

the head of the list and find the type of it.



The traversal is implemented (`Query.atPos`) but getting the type of an

arbitrary result is mostly not, except for the case of a variable in a `val`

binding, as a proof of concept.



The idea would be that these two APIs (location-based queries and query results

wrapping typed AST nodes) could be used to power IDE features like "show me the

type" or "go to definition"



## Gotchas / future work



- This only works for a single file. To do multiple files, it would require

  figuring out the APIs to use the Compilation Manager (CM) structures to load

  and type check a file in the context of a number of files and libraries.



- It currently prints out `?.` in front of all the types...



  Honestly, I think this has to do with the next gotcha, which is:



- It doesn't even typecheck a file in the context of the pervasive/basis

  library. Again, it's likely that integrating with CM would fix this?



The APIs **should** all be there, it's just not documented. When I was looking

into this last, I basically grepped for `elabTop` in the SML/NJ sources, found

one result in `base/compiler/TopLevel/main/compile.sml`, and was in the process

of working my way out from there to see how it sets up the environment that gets

fed into `elabTop`. Presumably all we have to do is mimic enough of that code

that it would work (or maybe there's an even higher level API we could call that

both takes care of setting up the right environment and gives us back an `Absyn`

typed AST).



## Understanding the Visible Compiler APIs



The online docs for the SML/NJ Visible Compiler APIs are terrible.



You're going to have much better luck searching through and reading the SML/NJ

source. It's important that you look at the source for the right version,

because these APIs tend to break in minor ways from version to version.



Some ways to get the sources for the version relevant to you:



-   If you know how to use SVN (or want to learn):



    ```

    svn checkout --username anonsvn --password anonsvn https://smlnj-gforge.cs.uchicago.edu/svn/smlnj/

    ```



    and then figure out how to checkout the version you're using



-   Otherwise, use the "install from source" instructions on the release page for

    your version of SML/NJ, for example:



    



    And be sure to follow the suggestion to edit `config/targets` so that you

    uncomment whatever is required to get the compiler source code downloaded

    too. For 110.99.2, this amounted to ensuring that this directive was

    uncommented in the `config/targets` file:



    ```

    # unpack the source code for everything (including for the SML/NJ compiler

    # itself); this is not required,  unless you are doing compiler hacking,

    # but it may be interesting to look at.

    #

    request src-smlnj

    ```



-   If you're really lazy, you can probably get by with using the SVN web viewer



    



    but it takes some getting used to, and you're very quickly want to be able

    to start grepping, so you probably only want to use this in a pinch.



Once you've got the sources, the `base/compiler` folder is the one where all the

good stuff is. If you chose an SVN option above (instead of downloading just the

sources for the given version), you'll have to make sure that you're in the

right `base/compiler` folder. For 110.99.2, that folder is

`/sml/releases/release-110.99.2/base/compiler`.



Once you have the SML/NJ sources, you should be able to work from the

dependencies in this example project's CM file to the relevant files in your

SML/NJ checkout to begin browsing what APIs are available to you.



[SML/NJ]: 

[Visible Compiler]: 

[symbol]: https://github.com/jez/symbol