{"id":16965022,"url":"https://github.com/propensive/superlunary","last_synced_at":"2025-04-11T23:02:43.586Z","repository":{"id":212685625,"uuid":"732066673","full_name":"propensive/superlunary","owner":"propensive","description":"Exploiting lightweight modular staging in Scala","archived":false,"fork":false,"pushed_at":"2025-02-11T23:54:42.000Z","size":2779,"stargazers_count":2,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-20T01:38:46.244Z","etag":null,"topics":["lms","scala","staging"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/propensive.png","metadata":{"files":{"readme":".github/readme.md","changelog":null,"contributing":".github/contributing.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-15T14:59:38.000Z","updated_at":"2025-02-11T23:54:45.000Z","dependencies_parsed_at":"2023-12-30T08:22:13.416Z","dependency_job_id":"fc3f7fc8-bae5-40d0-af07-8e50e313573e","html_url":"https://github.com/propensive/superlunary","commit_stats":null,"previous_names":["propensive/superlunary"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/propensive%2Fsuperlunary","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/propensive%2Fsuperlunary/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/propensive%2Fsuperlunary/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/propensive%2Fsuperlunary/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/propensive","download_url":"https://codeload.github.com/propensive/superlunary/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239805887,"owners_count":19700203,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["lms","scala","staging"],"created_at":"2024-10-13T23:44:50.680Z","updated_at":"2025-02-20T08:31:39.159Z","avatar_url":"https://github.com/propensive.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"[\u003cimg alt=\"GitHub Workflow\" src=\"https://img.shields.io/github/actions/workflow/status/propensive/superlunary/main.yml?style=for-the-badge\" height=\"24\"\u003e](https://github.com/propensive/superlunary/actions)\n[\u003cimg src=\"https://img.shields.io/discord/633198088311537684?color=8899f7\u0026label=DISCORD\u0026style=for-the-badge\" height=\"24\"\u003e](https://discord.com/invite/MBUrkTgMnA)\n\u003cimg src=\"/doc/images/github.png\" valign=\"middle\"\u003e\n\n# Superlunary\n\n__Exploiting lightweight modular staging__\n\nWhen we write a program in Scala, _usually_ all the code that is compiled\ntogether will be run together (along with dependencies), in the _same\nenvironment_: a single instance of the JVM, or in a browser with Scala.JS.\n(Macros are a notable exception, since they are run during a later compilation,\nbut they generally appear in _library_ rather than _application_ code.)\nConversely, in distributed applications, code which is intended to run in\n_different environments_ would be compiled separately, and would remain\nseparate from source code to its execution in separate JVMs, web browsers, and\nmaybe multiple machines or docker instances.\n\nExamples include client-server communications between an HTTP server and a web\nbrowser, and microservice-based architectures. For a distributed application,\nthere is an inherent contract between any two distinct environments in the\nsystem.\n\nHowever, our usual approach to development means that this contract is not\nenforced by the compiler. Even though other tools may be employed to ensure\ncontractual consistency, this must happen externally to the compiler, and\nrarely offers the same strong guarantees that Scala can. So contractual\nconsistency can be compromised, and lead to runtime failures.\n\n_Superlunary_'s model compiles source code to be run in different environments\n_together_, using _quotes and splices_ to precisely and safely delimit local\nfrom remote code. This allows code which runs in a remote environment to be\nwritten alongside local code; to be fully checked by the compiler; to be\nmarshalled and unmarshalled transparently; and to be maintained in lockstep.\n\n_Superlunary_ makes it possible to develop a distributed application with the\nversatility, simplicity and self-consistency as an application which runs\nwithin a single runtime environment.\n\n## Features\n\n- write remote code in-place using quotes and splices\n- embed local values seamlessly into remote code\n- lightweight, yet clearly-delimited code\n- compile code using a custom compiler, to be run remotely\n- ensure contractual consistency between local and remote code\n- widespread applications in many microservice and client-server scenarios\n\n\n\n## Availability\n\n\n\n\n\n\n\n## Getting Started\n\n### Background\n\n#### Quotes and Splices\n\nA Scala 3 macro, written using quotes and splices syntax, typically looks something like this:\n```scala\nimport scala.quoted.*\n\ndef say(user: String)(using Quotes): Expr[Unit] =\n  val name: Expr[String] = Expr(user)\n  '{Out.println(\"Hello \"+${name})}\n```\n\nThe usage of `'{...}` (quotes) and `${...}` (splices) are indicative of _phase\nshifts_.  Code inside quotes will be executed one phase later than the\nsurrounding code, and code inside a splice will be executed one phase earlier\nthan the code surrounding it. In the example above, the definition of `name`\nand the usage of `name` occur in the same phase (and *must* occur in the same\nphase, due to the _Phase Consistency Principle_), while `Out.println` and\n`\"Hello \"` are in the next phase. An instance of `Expr[String]` or `Expr[Unit]`\nis an abstract notion of an expression that will become a `String` or a `Unit`\nin the next phase.\n\nFor a macro, that \"next phase\" will be a later compilation, when a new instance\nof the compiler is run, and all code from prior phases exists as compiled\nbytecode, and can be run.\n\nA similar effect could be achieved just by writing the code in a separate file,\nand compiling it later, but the clever part is that quotes and splices can be\ninterleaved at a fine-grained level with multiple levelse of nesting; as\nexpressions. And furthermore, those expressions are typechecked for\n_consistency across phases_.\n\nBut quotes and splices and the concept of phases can be applied more generally\nthan in plain macros. The \"next phase\" does not have to be \"the next\ncompilation\"; a quoted block can represent any code which runs \"elsewhere\".\nThere are a world of possibilities for where \"elsewhere\" could be: it could be\nin another JVM, on a cloud-based system, or inside a browser using ScalaJS.\n\n_Superlunary_ provides the wiring to make it easy to exploit the powerful\nsyntax and consistency-checking of quoted code, to make it possible to write\ncode with seamless syntax which can be dispatched to run in an environment of\nyour choosing, with very little effort.\n\n### Usage\n\n_Superlunary_ provides the most general mechanism to make it possible to\nremotely run code which is written in an inline style. The library could be\nemployed in a wide variety of different projects.\n\nOther libraries may use _Superlunary_ to provide wiring for different remote\nenvironments. (Note that we will generally use \"remote\" to mean \"external to\nthe current JVM\". It may return to a different physical computer accessed\nacross a network, or may not.) These would typically be its direct dependents,\nand would implement the logic necessary to package and deploy arbitrary code to\nthe environment in which it would be run, spawn the environment, provide its\ninput parameters, launch it and capture its result. These implementations would\nbe very different for code being deployed to a web browser, compared to code\ndeployed to a docker container or cloud virtual machine.\n\nThese libraries will be called _dispatch providers_ and implement _dispatchers_.\n\nFurther libraries (or applications) may make use of one or more dispatchers\nas a convenient way to run their code in a different environment, and\ncould be maintained by entirely different developers. Libraries like this will\nneed to implement that code using _quotes and splices_ syntax, but will present\nit to dependents as ordinary methods, like any other API.\n\nThese libraries will be called _dispatch clients_.\n\nAny downstream libraries may call these methods, blissfully unaware that\n_Superlunary_ is involved.\n\nThe two \"interesting\" uses of _Superlunary_ are dispatch providers and\ndispatch clients.\n\n### Writing a Dispatch Client\n\nAll terms and types are defined in the `superlunary` package:\n```scala\nimport superlunary.*\n```\n\nWriting a dispatch client presumes we have a dispatcher, which may come from a\nthird-party library, or may be defined locally. For now, we will assume that we\nhave a dispatcher, an instance of `Dispatcher`, which will dispatch some\ncode _somewhere else_ to run. `Dispatcher` is designed so that the exact\nmeaning of \"somewhere else\" does not affect the way it is used, and can remain\nabstract.\n\nThus, given a `Dispatcher` object, called `Remote`, we can call its 'dispatch'\nmethod, passing in a quoted block of code, like so:\n```scala\ndef run(): Unit =\n  Remote.dispatch:\n    '{ println(\"Hello world\") }\n```\n\nInvoking `run()` will dispatch the code `println(\"Hello world\")` to its remote\nenvironment, run it, and return the result. The result is just the `Unit` value,\nso it's not very interesting. And unless there's a console connected to the\nremote environment, we won't be able to see the words, `Hello world`.\n\nBut if it returns without an exception, then we can assume it executed\nsuccessfully.\n\nA more interesting example could return a value. The\n[Inimitable](https://github.com/propensive/inimitable/) library provides a\nmethod to return a UUID corresponding to the currently running JVM instance,\nwhich remains static for the lifetime of the JVM. We can use this to check that\nthe code is running on a different JVM:\n```scala\nimport inimitable.*\n\ndef run(): Unit =\n  val remoteUuid: Uuid = Remote.dispatch('{jvmInstanceId})\n  val localUuid: Uuid = jvmInstanceId\n  println(t\"$remoteUuid vs $localUuid\")\n```\n\nHere, the same invocation is called twice: once remotely, and once locally, and\nwe print both for comparison. We should see two different values.\n\nNote a significant change in the `Remote.dispatch` call: it is now returning a\n`Uuid`, rather than `Unit`. This allows us to get a value back from the remote\nJVM. Note also how simple the remote code is—a single expression—and that we\nare able to access it without a prefix, because the `inimitable` import outside\nof the quotes is sufficient for its name to be resolved.\n\nThat is only true for _static prefixes_: those names which can be resolved\nstatically. It would not make sense to be able to access heap references which\nexist within the memory of the local JVM, but have no meaning on the remote.\nAnd this is exactly the protection that the _phase consistency principle_ gives\nus.\n\n#### Marshalling\n\nThe remote code defined inside the `dispatch` call returns a `Uuid`, which (at\nthat point) will exist only as a heap value in the remote JVM. In order for it\nto exist within the local JVM, it must be transmitted by some means. There are\nmany ways this _could_ happen, but currently _Superlunary_ uses\n[Jacinta](https://github.com/propensive/jacinta/) to serialize and deserialize\nvalues to and from JSON strings.\n\nThis is possible for any type that has both a `JsonEncoder` and `JsonDecoder`\ntypeclass instance, and it happens transparently, entirely _behind the scenes_.\nAs long as the necessary typeclass instances exist (or can be automatically\nconstructed by generic derivation), _Superlunary_ will apply the encoding and\ndecoding wherever necessary.\n\n#### Providing inputs\n\nIt's possible to pass local values into the remote code, much as values are\nsubstituted into an interpolated string, or (even moreso) spliced into a macro.\n\nThe only requirement is that values be spliced as `Expr`s. That is, if `x` is\nan `Int`, we must splice it as an `Expr[Int]`, which is trivially possible by\ncalling the `put` extension method on it.\n\nFor example, we can check the lag between local and remote execution with a\n`lag` method:\n```scala\ndef lag(currentTime: Long): Long = remote.dispatch:\n  '{System.currentTimeMillis - ${currentTime.put}}\n```\nor even more directly:\n```scala\ndef lag(): Long = remote.dispatch:\n  '{System.currentTimeMillis - ${System.currentTimeMillis.put}}\n```\n\nAs with return parameters, all marshalling to and from JSON is handled\nautomatically by _Superlunary_.\n\n### Writing a `Dispatcher`\n\nA dispatch provider will provide a singleton instance of `Dispatcher`, or some\nmeans of constructing new `Dispatcher`s, potentially with parameters which\ndetermine the remote environment.\n\nThe definition of `Dispatcher` is relatively simple. It provides an\nimplementation of `dispatch`—the method which dispatch clients will call—and\nrequires a simpler method, `invoke`, to be implemented. Additionally, the type\nconstructor member, `Result[OutputType]`, should be specified to determine how\nthe raw return type of the dispatched code (`OutputType`) should be transformed\ninto a result from executing it.\n\nAdditionally, its `scalac` value is a specification for the invocation of the\nScala compiler, as specified in\n[Anthology](https://github.com/propensive/anthology/).\n\nFor example, we could return an `Optional[OutputType]`, an `Async[OutputType]`\nor simply `OutputType` itself. Or if we don't care about the result, we could\neven return `Unit`.\n\nThe signature of `invoke` looks like this:\n```scala\nprotected def invoke[OutputType](dispatch: Dispatch[OutputType]): Result[OutputType]\n```\n\nWe need to implement it to produce a `Result[OutputType]`, using the input\ninformation that has been packaged together in the `Dispatch[OutputType]`\nvalue, `dispatch`. A `Dispatch` value provides the following:\n - `path`, a `Path` of the output from compiling the dispatched code\n - `classpath`, the full `LocalClasspath` that was used for compilation\n - `mainClass`, the name of the class whose `main` method should be invoked\n - `local`, a function value of `() =\u003e OutputType` which invokes the code\n   locally; included for completeness\n - `remote`, a function value of `(Text =\u003e Text) =\u003e OutputType`, which will\n   form the crux of the implementation\n\nTogether, these values can be used to implement a `Dispatcher` instance.\n\nWhen a user calls `dispatch` on a `Dispatcher`, several tasks are performed in\nconstructing a `Dispatch` value before delegating to the `Dispatcher`'s\n`invoke` method. These include:\n - extracting the classpath from the current classloader\n - compiling the quoted code, if it has not already been compiled\n - capturing the spliced input values\n - encoding the inputs as a single JSON value, which is serialized as `Text`\n\nThis reduces the implementation of `invoke` to a simpler core task: to execute\nthe `main` method of a particular class from a provided classpath (in an\nenvironment of our choosing), passing in a single `Text` value, and returning\nthe `Text` value that results from calling that method.\n\nThis is how it works in practice:\n```scala\nobject Remote extends Dispatcher:\n  type Result[OutputType] = Optional[ResultType]\n\n  protected val scalac = Scalac[3.4](Nil)\n\n  protected def invoke(dispatch: Dispatch): Optional[ResultType] =\n    dispatch.remote { input =\u003e executeRemotely(input) }\n```\n\nSo we just need to specify what `executeRemotely` should do.\n\nAs an example, we will using Guillotine to run the `java` command and launch a\nnew JVM locally. The shell command we need to run will look similar to this:\n```scala\njava -classpath \u003cclasspath\u003e \u003cmain-class\u003e \u003cinput\u003e\n```\n\nWe will use [Guillotine](https://github.com/propensive/guillotine/) to run that\nshell command:\n```scala\ndef invoke(dispatch: Dispatch): Optional[ResultType] =\n  dispatch.remote: input =\u003e\n    val command = sh\"java -classpath ${dispatch.classpath()} ${dispatch.mainClass} $input\"\n    command.exec[Text]()\n```\n\nThe `command` definition specifies the command to be run, and the final line,\n`command.exec[Text]()`, runs it and captures its standard output as a single\n`Text` value. We could print the `command` value before executing it, to check\nexactly what will be run.\n\nThe `mainClass` value is actually a fixed value,\n`\"superlunary.DispatchRunner\"`, but it is provided as a named value so that\nthat exact name is not part of the public API, in case it ever changes.\n`DispatchRunner` itself provides the `main` method we used above, which is\nsuitable for running the class from outside a JVM. As with all `main` methods,\nthis takes an array of strings as input and returns no value—so it _prints_ the\nreturn value, which we capture above.\n\nThis is not so convenient for performing the executing from _within_ a JVM, so\na `run` method is also provided, which takes a single `String` parameter and\nreturns a `String`. This offers a useful alternative if reflection is used to\ninvoke the code.\n\nThe example above allows such a simple implementation in part because it runs\non the same machine and the references to directories and JAR files specified\nin the `classpath` value are readily available on the same machine. However,\nany (genuinely) remote execution will need those class files to be made\navailable in the remote environment.\n\nThe `classpath` is an immutable value whose `entries` value can be used as a\nmeans to get hold of these files and directories for remote deployment. But\nmore specifically, the `path` value provides the location (within a temporary\ndirectory) of the newly-compiled source files.\n\n### Caching\n\nThe remote quoted code will be compiled within the running JVM the first time\nit is encountered, and subsequent invocations will reuse the cached version.\n\n\n## Status\n\nSuperlunary is classified as __embryotic__. For reference, Soundness projects are\ncategorized into one of the following five stability levels:\n\n- _embryonic_: for experimental or demonstrative purposes only, without any guarantees of longevity\n- _fledgling_: of proven utility, seeking contributions, but liable to significant redesigns\n- _maturescent_: major design decisions broady settled, seeking probatory adoption and refinement\n- _dependable_: production-ready, subject to controlled ongoing maintenance and enhancement; tagged as version `1.0.0` or later\n- _adamantine_: proven, reliable and production-ready, with no further breaking changes ever anticipated\n\nProjects at any stability level, even _embryonic_ projects, can still be used,\nas long as caution is taken to avoid a mismatch between the project's stability\nlevel and the required stability and maintainability of your own project.\n\nSuperlunary is designed to be _small_. Its entire source code currently consists\nof 184 lines of code.\n\n## Building\n\nSuperlunary will ultimately be built by Fury, when it is published. In the\nmeantime, two possibilities are offered, however they are acknowledged to be\nfragile, inadequately tested, and unsuitable for anything more than\nexperimentation. They are provided only for the necessity of providing _some_\nanswer to the question, \"how can I try Superlunary?\".\n\n1. *Copy the sources into your own project*\n   \n   Read the `fury` file in the repository root to understand Superlunary's build\n   structure, dependencies and source location; the file format should be short\n   and quite intuitive. Copy the sources into a source directory in your own\n   project, then repeat (recursively) for each of the dependencies.\n\n   The sources are compiled against the latest nightly release of Scala 3.\n   There should be no problem to compile the project together with all of its\n   dependencies in a single compilation.\n\n2. *Build with [Wrath](https://github.com/propensive/wrath/)*\n\n   Wrath is a bootstrapping script for building Superlunary and other projects in\n   the absence of a fully-featured build tool. It is designed to read the `fury`\n   file in the project directory, and produce a collection of JAR files which can\n   be added to a classpath, by compiling the project and all of its dependencies,\n   including the Scala compiler itself.\n   \n   Download the latest version of\n   [`wrath`](https://github.com/propensive/wrath/releases/latest), make it\n   executable, and add it to your path, for example by copying it to\n   `/usr/local/bin/`.\n\n   Clone this repository inside an empty directory, so that the build can\n   safely make clones of repositories it depends on as _peers_ of `superlunary`.\n   Run `wrath -F` in the repository root. This will download and compile the\n   latest version of Scala, as well as all of Superlunary's dependencies.\n\n   If the build was successful, the compiled JAR files can be found in the\n   `.wrath/dist` directory.\n\n## Contributing\n\nContributors to Superlunary are welcome and encouraged. New contributors may like\nto look for issues marked\n[beginner](https://github.com/propensive/superlunary/labels/beginner).\n\nWe suggest that all contributors read the [Contributing\nGuide](/contributing.md) to make the process of contributing to Superlunary\neasier.\n\nPlease __do not__ contact project maintainers privately with questions unless\nthere is a good reason to keep them private. While it can be tempting to\nrepsond to such questions, private answers cannot be shared with a wider\naudience, and it can result in duplication of effort.\n\n## Author\n\nSuperlunary was designed and developed by Jon Pretty, and commercial support and\ntraining on all aspects of Scala 3 is available from [Propensive\nO\u0026Uuml;](https://propensive.com/).\n\n\n\n## Name\n\nThat which is _superlunary_ (by contrast to _sublunary_), is literally \"beyond the moon\", and hence \"otherworldly\", and describes the code in quotes which Superlunary enables to run elsewhere.\n\nIn general, Soundness project names are always chosen with some rationale,\nhowever it is usually frivolous. Each name is chosen for more for its\n_uniqueness_ and _intrigue_ than its concision or catchiness, and there is no\nbias towards names with positive or \"nice\" meanings—since many of the libraries\nperform some quite unpleasant tasks.\n\nNames should be English words, though many are obscure or archaic, and it\nshould be noted how willingly English adopts foreign words. Names are generally\nof Greek or Latin origin, and have often arrived in English via a romance\nlanguage.\n\n## Logo\n\nThe logo shows a moon, reflected in water.\n\n## License\n\nSuperlunary is copyright \u0026copy; 2025 Jon Pretty \u0026 Propensive O\u0026Uuml;, and\nis made available under the [Apache 2.0 License](/license.md).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpropensive%2Fsuperlunary","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpropensive%2Fsuperlunary","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpropensive%2Fsuperlunary/lists"}