{"id":13580981,"url":"https://github.com/databricks/sjsonnet","last_synced_at":"2025-05-16T01:04:10.458Z","repository":{"id":33959663,"uuid":"137439053","full_name":"databricks/sjsonnet","owner":"databricks","description":null,"archived":false,"fork":false,"pushed_at":"2025-05-11T03:19:29.000Z","size":1749,"stargazers_count":281,"open_issues_count":25,"forks_count":59,"subscribers_count":250,"default_branch":"master","last_synced_at":"2025-05-11T03:30:23.965Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/databricks.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-06-15T03:57:54.000Z","updated_at":"2025-05-11T03:19:29.000Z","dependencies_parsed_at":"2022-08-07T23:31:00.280Z","dependency_job_id":"f78b2f54-f581-4ae4-96df-a7bb29818aef","html_url":"https://github.com/databricks/sjsonnet","commit_stats":{"total_commits":526,"total_committers":42,"mean_commits":"12.523809523809524","dds":0.7262357414448669,"last_synced_commit":"759cea713bc8e39f45b844a75358fd780f75d480"},"previous_names":["lihaoyi/sjsonnet"],"tags_count":44,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fsjsonnet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fsjsonnet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fsjsonnet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fsjsonnet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/databricks","download_url":"https://codeload.github.com/databricks/sjsonnet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254448579,"owners_count":22072764,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T15:01:56.980Z","updated_at":"2025-05-16T01:04:10.449Z","avatar_url":"https://github.com/databricks.png","language":"Scala","readme":"# Sjsonnet\nA Scala implementation of the [Jsonnet](https://jsonnet.org/) configuration language, running on JVM, Scala Native and JavaScript.\n\n## Usage\n\nSjsonnet can be used from Java and Scala:\n\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.databricks\u003c/groupId\u003e\n    \u003cartifactId\u003esjsonnet_3\u003c/artifactId\u003e\n\u003c/dependency\u003e\n```\n\n```java\n// Java\nsjsonnet.SjsonnetMain.main0(\n    new String[]{\"foo.jsonnet\"},\n    new DefaultParseCache,\n    System.in,\n    System.out,\n    System.err,\n    os.package$.MODULE$.pwd(),\n    scala.None$.empty()\n);\n\n// Scala\nsjsonnet.SjsonnetMain.main0(\n    Array(\"foo.jsonnet\"),\n    new DefaultParseCache,\n    System.in,\n    System.out,\n    System.err,\n    os.pwd, // working directory\n    None\n);\n```\n\nAs a standalone executable assembly from the [github release page](https://github.com/databricks/sjsonnet/releases):\n```bash\n$ chmod +x sjsonnet.jar\n\n$ ./sjsonnet.jar\nerror: Need to pass in a jsonnet file to evaluate\nusage: sjsonnet [sjsonnet-options] script-file\n\n  -n, --indent       How much to indent your output JSON\n  -J, --jpath        Specify an additional library search dir (left-most wins)\n  -o, --output-file  Write to the output file rather than stdout\n  ...\n\n$ ./sjsonnet.jar foo.jsonnet\n```\n\nOr from Javascript:\n\n```javascript\n$ node\n\n\u003e require(\"./sjsonnet.js\")\n\n\u003e SjsonnetMain.interpret(\"local f = function(x) x * x; f(11)\", {}, {}, \"\", (wd, imported) =\u003e null)\n121\n\n\u003e SjsonnetMain.interpret(\n    \"local f = import 'foo'; f + 'bar'\", // code\n    {}, // extVars\n    {}, // tlaVars\n    \"\", // initial working directory\n\n    // import callback: receives a base directory and the imported path string,\n    // returns a tuple of the resolved file path and file contents or file contents resolve method\n    (wd, imported) =\u003e [wd + \"/\" + imported, \"local bar = 123; bar + bar\"],\n    // loader callback: receives the tuple from the import callback and returns the file contents\n    ([path, content]) =\u003e content\n    )\n'246bar'\n```\n\nNote that since Javascript does not necessarily have access to the\nfilesystem, you have to provide an explicit import callback that you can\nuse to resolve imports yourself (whether through Node's `fs` module, or\nby emulating a filesystem in-memory)\n\n### Running deeply recursive Jsonnet programs\n\nThe depth of recursion is limited by running environment stack size. You can run Sjsonnet with increased\nstack size as follows:\n\n```bash\n# JVM\njava -Xss100m -cp sjsonnet.jar sjsonnet.SjsonnetMain foo.jsonnet\n\n# Scala Native\nSCALANATIVE_THREAD_STACK_SIZE=100m ./sjsonnet foo.jsonnet\n\n# ScalaJS (Node)\nnode --stack-size=100m\n```\n\n## Architecture\n\nSjsonnet is implementated as an optimizing interpreter. There are roughly 4\nphases:\n\n- `sjsonnet.Parser`: parses an input `String` into a `sjsonnet.Expr`, which is a\n  [Syntax Tree](https://en.wikipedia.org/wiki/Abstract_syntax_tree) representing\n  the Jsonnet document syntax, using the\n  [Fastparse](https://github.com/lihaoyi/fastparse) parsing library\n\n- `sjsonnet.StaticOptimizer` is a single AST transform that performs static\n  checking, essential rewriting (e.g. assigning indices in the symbol table for\n  variables) and optimizations. The result is another `sjsonnet.Expr` per input\n  file that can be stored in the parse cache and reused.\n\n- `sjsonnet.Evaluator`: recurses over the `sjsonnet.Expr` produced by the\n  optimizer and converts it into a `sjsonnet.Val`, a data structure representing\n  the Jsonnet runtime values (basically lazy JSON which can contain function values).\n\n- `sjsonnet.Materializer`: recurses over the `sjsonnet.Val` and converts it into\n  an output `ujson.Expr`: a non-lazy JSON structure without any remaining\n  un-evaluated function values. This can be serialized to a string formatted in a\n  variety of ways\n\nThese three phases are encapsulated in the `sjsonnet.Interpreter` object.\n\nSome notes on the values used in parts of the pipeline:\n\n- `sjsonnet.Expr`: this represents `{...}` object literal nodes, `a + b` binary\n  operation nodes, `function(a) {...}` definitions and `f(a)` invocations, etc..\n  Also keeps track of source-offset information so failures can be correlated\n  with line numbers.\n\n- `sjsonnet.Val`: essentially the JSON structure (objects, arrays, primitives)\n  but with two modifications. The first is that functions like\n  `function(a){...}` can still be present in the structure: in Jsonnet you can\n  pass around functions as values and call then later on. The second is that\n  object values \u0026 array entries are _lazy_: e.g. `[error 123, 456][1]` does not\n  raise an error because the first (erroneous) entry of the array is un-used and\n  thus not evaluated.\n\n- Classes representing literals extend `sjsonnet.Val.Literal` which in turn extends\n  _both_, `Expr` and `Val`. This allows the evaluator to skip over them instead of\n  having to convert them from one representation to the other.\n\n## Performance\n\nDue to pervasive caching, sjsonnet is much faster than google/jsonnet. See\nthis blog post for more details:\n\n- [Writing a Faster Jsonnet Compiler](https://databricks.com/blog/2018/10/12/writing-a-faster-jsonnet-compiler.html)\n\nHere's the latest set of benchmarks I've run (as  of 18 May 2023) comparing Sjsonnet against\ngoogle/go-jsonnet and google/jsonnet, measuring the time taken to\nevaluate an arbitrary config file in the Databricks codebase:\n\n|              | Sjsonnet 0.4.3 | google/go-jsonnet 0.20.0 | google/jsonnet 0.20.0 |\n| :----------- | -------------: | -------------: | -------------: |\n| staging/runbot-app.jsonnet (~6.6mb output JSON) | ~0.10s |  ~6.5s | ~67s |\n\nSjsonnet was run as a long-lived daemon to keep the JVM warm,\nwhile go-jsonnet and google/jsonnet were run as subprocesses, following typical\nusage patterns. The Sjsonnet command\nline which is run by all of these is defined in\n`MainBenchmark.mainArgs`. You need to change it to point to a suitable input\nbefore running a benchmark or the profiler.\n\nBenchmark example:\n\n```\nsbt bench/jmh:run -jvmArgs \"-XX:+UseStringDeduplication\" sjsonnet.MainBenchmark\n```\n\nProfiler:\n\n```\nsbt bench/run\n```\n\nThere's also a benchmark for memory usage:\n\nExecute and print stats:\n```\nsbt 'set fork in run := true' 'set javaOptions in run ++= Seq(\"-Xmx6G\", \"-XX:+UseG1GC\")' 'bench/runMain sjsonnet.MemoryBenchmark'\n```\n\nExecute and pause - this is useful if you want to attach a profiler after the run and deep dive the\nobject utilization.\n```\nsbt 'set fork in run := true' 'set javaOptions in run ++= Seq(\"-Xmx6G\", \"-XX:+UseG1GC\")' 'bench/runMain sjsonnet.MemoryBenchmark --pause'\n```\n\n## Laziness\n\nThe Jsonnet language is _lazy_: expressions don't get evaluated unless\ntheir value is needed, and thus even erroneous expressions do not cause\na failure if un-used. This is represented in the Sjsonnet codebase by\n`sjsonnet.Lazy`: a wrapper type that encapsulates an arbitrary\ncomputation that returns a `sjsonnet.Val`.\n\n`sjsonnet.Lazy` is used in several places, representing where\nlaziness is present in the language:\n\n- Inside `sjsonnet.Scope`, representing local variable name bindings\n\n- Inside `sjsonnet.Val.Arr`, representing the contents of array cells\n\n- Inside `sjsonnet.Val.Obj`, representing the contents of object values\n\n`Val` extends `Lazy` so that an already computed value can be treated as\nlazy without having to wrap it.\n\nUnlike [google/jsonnet](https://github.com/google/jsonnet), Sjsonnet caches the\nresults of lazy computations the first time they are evaluated, avoiding\nwasteful re-computation when a value is used more than once.\n\n## Standard Library\n\nDifferent from [google/jsonnet](https://github.com/google/jsonnet), Sjsonnet\ndoes not implement the Jsonnet standard library `std` in Jsonnet code. Rather,\nthose functions are implemented as intrinsics directly in the host language (in\n`Std.scala`). This allows both better error messages when the input types are\nwrong, as well as better performance for the more computationally-intense\nbuiltin functions.\n\n## Client-Server\n\nSjsonnet comes with a built in thin-client and background server, to help\nmitigate the unfortunate JVM warmup overhead that adds ~1s to every invocation\ndown to 0.2-0.3s. For the simple non-client-server executable, you can use\n\n```bash\n./mill -i show sjsonnet[3.3.6].jvm.assembly\n```\n\nTo create the executable. For the client-server executable, you can use\n\n```bash\n./mill -i show sjsonnet[3.3.6].server.assembly\n```\n\nBy default, the Sjsonnet background server lives in `~/.sjsonnet`, and lasts 5\nminutes before shutting itself when inactive.\n\nSince the Sjsonnet client still has 0.2-0.3s of overhead, if using Sjsonnet\nheavily it is still better to include it in your JVM classpath and invoke it\nprogrammatically via `new Interpreter(...).interpret(...)`.\n\n## Publishing\n\nTo publish the JVM version to Maven, make sure the version number in `build.sc` is correct, then run the following commands:\n```bash\n./mill -i mill.scalalib.PublishModule/publishAll \\\n    --sonatypeCreds $SONATYPE_USER:$SONATYPE_PASSWORD --publishArtifacts \"sjsonnet.jvm[_].__.publishArtifacts\" --release true \\\n    --gpgArgs --passphrase=$GPG_PASSPHRASE,--batch,--yes,-a,-b,--pinentry-mode=loopback\n```","funding_links":[],"categories":["Scala"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks%2Fsjsonnet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabricks%2Fsjsonnet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks%2Fsjsonnet/lists"}