{"id":15903121,"url":"https://github.com/can-lehmann/exprgrad","last_synced_at":"2025-07-18T12:36:12.994Z","repository":{"id":43421715,"uuid":"414133365","full_name":"can-lehmann/exprgrad","owner":"can-lehmann","description":"An experimental deep learning framework for Nim based on a differentiable array programming language","archived":false,"fork":false,"pushed_at":"2022-12-30T17:54:04.000Z","size":310,"stargazers_count":120,"open_issues_count":0,"forks_count":1,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-03-28T10:01:35.540Z","etag":null,"topics":["autodiff","automatic-differentiation","compiler","deep-learning","dsl","gradient","llvm","machine-learning","ml","neural-network","nim","opencl","tensor"],"latest_commit_sha":null,"homepage":"","language":"Nim","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/can-lehmann.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-10-06T08:38:44.000Z","updated_at":"2024-12-29T12:49:34.000Z","dependencies_parsed_at":"2023-01-31T12:30:44.118Z","dependency_job_id":null,"html_url":"https://github.com/can-lehmann/exprgrad","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/can-lehmann%2Fexprgrad","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/can-lehmann%2Fexprgrad/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/can-lehmann%2Fexprgrad/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/can-lehmann%2Fexprgrad/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/can-lehmann","download_url":"https://codeload.github.com/can-lehmann/exprgrad/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248968741,"owners_count":21191158,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autodiff","automatic-differentiation","compiler","deep-learning","dsl","gradient","llvm","machine-learning","ml","neural-network","nim","opencl","tensor"],"created_at":"2024-10-06T12:01:02.574Z","updated_at":"2025-04-14T21:52:10.746Z","avatar_url":"https://github.com/can-lehmann.png","language":"Nim","readme":"# Exprgrad\n\nExprgrad is an experimental deep learning framework for Nim based on a differentiable array programming language.\nExprgrad makes creating and training neural networks easy: \n\n```nim\nimport std/random\nimport exprgrad, exprgrad/layers/[base, dnn]\nrandomize(10)\n\nlet\n  net = input(\"x\")\n    .dense(2, 4).leakyRelu()  # 1st Layer\n    .dense(4, 1).sigmoid()    # 2nd Layer\n    .target(\"predict\")\n    .mse(input(\"y\"))          # Loss\n    .target(\"loss\")\n    .backprop(gradientDescent.makeOpt(rate=0.1)) # Train\n    .target(\"train\")\n  model = compile[float32](net)\n\nlet\n  trainX = Tensor.new([4, 2], @[float32 0, 0, 0, 1, 1, 0, 1, 1])\n  trainY = Tensor.new([4, 1], @[float32 0, 1, 1, 0])\n\nfor epoch in 0..\u003c5000:\n  model.apply(\"train\", {\"x\": trainX, \"y\": trainY})\n\necho model.call(\"predict\", {\"x\": trainX})\n```\n\nBecause exprgrad is based on a custom differentiable programming language, we do not need to rely on its built in layers.\nInstead we can also specify the same model in terms of scalar operations on tensors.\n\n```nim\n# Layer 1\nhidden*[y, x] ++= input(\"x\")[y, it] * param([2, 4])[it, x] | (y, x, it)\nhidden[y, x] ++= param([4])[x] | (y, x)\nhiddenRelu*{it} ++= select(hidden{it} \u003c= 0.0, 0.1 * hidden{it}, hidden{it}) | it\n# Layer 2\noutput*[y, x] ++= hiddenRelu[y, it] * param([4, 1])[it, x] | (y, x, it)\noutput[y, x] ++= param([1])[x] | (y, x)\noutputSigmoid*{it} ++= 1.0 / (1.0 + exp(-output{it})) | it\nlet pred = outputSigmoid.target(\"predict\")\nloss*[0] ++= sq(pred{it} - input(\"y\"){it}) | it # Loss\n\nproc optim(param: var Fun, grad: Fun) =\n  param{it} ++= -0.1 * grad{it} | it\n\nlet net = loss.target(\"loss\").backprop(optim).target(\"train\") # Train\n\nlet model = compile[float32](net)\n```\n\nSince exprgrad's compiler is able to derive any program written in its domain specific language, we do not need to specify a backwards pass.\nThis allows you to iterate on custom layers quickly, while avoiding errors in the gradient computation.\nThe model is optimized and compiled using a JIT compiler, enabling fast execution times.\nAll layers provided by exprgrad are also implemented in the same way, allowing you to customize them easily.\n\n## Installation\n\n**Warning:** Exprgrad is still very early in its development.\nAlthough all shown examples already work, bugs are expected and important features for training large models (especially Multithreading and GPU support) might still be missing.\nPlease report any issues you might encounter.\n\n### Ubuntu\n\n```bash\n$ sudo apt install llvm-13-dev\n$ nimble install exprgrad\n```\n\n**Note:** Your version of Ubuntu may not have the `llvm-13-dev` package in its repositories.\nFollow the instructions at [apt.llvm.org](https://apt.llvm.org/) to install the required repository.\n\n### Fedora 36\n\n```bash\n$ sudo dnf install llvm13-devel\n$ nimble install exprgrad\n```\n\n### Fedora 35\n\n```bash\n$ sudo dnf install llvm-devel\n$ nimble install exprgrad\n```\n\n## Documentation\n\n### Language\n\nExprgrad's custom differentiable array programming language is used to specify all layers.\nIt is a custom language which differs greatly from Nim both in syntax and semantics.\nKernels/layers written in exprgrad's language are embedded in Nim programs and created using the `++=` macro.\n\nThe language does not have functions, procedures or structured control flow.\nInstead each program is a single expression inside a series of implicitly specified nested loops.\nA simple program which multiplies two matrices looks like this:\n\n```nim\nproc matmul(a, b: Fun): Fun =\n  result[y, x] ++= a[y, it] * b[it, x] | (y, x, it)\n```\n\nThe same program in Nim would look like this:\n\n```nim\nproc `*`*[T](a, b: Tensor[T]): Tensor[T] =\n  result = Tensor[T].new([a.shape[0], b.shape[1]])\n  for y in 0..\u003cresult.shape[0]:\n    for it in 0..\u003ca.shape[1]:\n      for x in 0..\u003cresult.shape[1]:\n        result[y, x] += a[y, it] * b[it, x]\n```\n\nAs you can see, the program in exprgrad's domain-specific language is basically equivalent to the last line of the Nim program.\nThe shape of the output tensor and the iteration ranges of all loops are inferred automatically.\n\nIn contrast to Nim, exprgrad's type system is very simple as it includes only four types.\n\n| Name       | Purpose                                            |\n| ---------- | -------------------------------------------------- |\n| `Scalar`   | Floating point value. Is differentiable.           |\n| `Index`    | Integer value. Used to index into tensors.         |\n| `Boolean`  | Boolean value. Only used in `select` instructions. |\n| `Array[T]` | Fixed size array with items of type T.             |\n\nTensors may be accessed using the `[]` and `{}` operators.\nWhile `[]` allows you index into each dimension, `{}` gives you direct access to the data of the tensor.\nBecause `{}` does not allow exprgrad to infer tensor shapes in all cases, `[]` should always be preferred over `{}`. \n\n```nim\nproc identity*(a: Fun): Fun =\n  result{it} ++= a{it} | it\n```\n\nLiterals for each type are available.\nNote that exprgrad does not have automatic type conversions.\n`Scalar` literals therefore must include a point (`2.0` instead of `2`) to differentiate them from `Index` literals.\n\n```nim\nproc double*(a: Fun): Fun =\n  result{it} ++= a{it} * 2.0 | it\n```\n\nVariables from Nim may be included as static values.\nOnly variables of type `int`, `float64` and `bool` can be included.\n\n```nim\nproc `*`*(a: Fun, factor: float64): Fun =\n  result{it} ++= a{it} * factor | it\n```\n\nConditionals can be emulated using the `select` instruction.\nThere is no guarantee that both branches are executed.\n\n```nim\nproc relu*(inp: Fun): Fun =\n  result{it} ++= select(inp{it} \u003e= 0.0, inp{it}, 0.0) | it\n```\n\nAn expression may contain multiple statements separated using `;`.\nThis allows you to define variables using the `let` statement and use them later on.\n\n```nim\nproc tanh*(inp: Fun): Fun =\n  result{it} ++= (\n    let a = exp(inp{it});\n    let b = exp(-inp{it});\n    (a - b) / (a + b)\n  ) | it\n```\n\nIf exprgrad is not able to infer the shape of a tensor, it can be explicitly specified using `withShape` or `copyShape`.\nThe argument to the `withShape` macro must be of the form `[dim0, dim1, dim2, ...]` where each dimension is a valid expression in exprgrad's language.\n\n```nim\nproc upsample2*(images: Fun): Fun =\n  result[image, y, x, chan] ++= images[image, y div 2, x div 2, chan] | (image, y, x, chan)\n  result.withShape([\n    images.shape[0],\n    images.shape[1] * 2,\n    images.shape[2] * 2,\n    images.shape[3]\n  ])\n```\n\nIf the output tensor is not yet declared, the `*` operator can be added after the tensor's name to declare it.\n\n```nim\ny*{it} ++= input(\"x\"){it} * 2.0 | it\n```\n\nSometimes you might want to use a custom gradient implementation instead of the one automatically generated by exprgrad.\nThis is especially useful for ensuring numerical stability or improving performance.\nInside the `customGrad` attribute, gradient tensors are referred to using the `grad(tensor)` instruction.\n\n```nim\nidentity*{x} ++= inp{x} | x do:\n  customGrad:\n    grad(inp){x} ++= inp{x} * grad(identity){x} | x\n```\n\nMore examples can be found in the `exprgrad/layers/base.nim` and `exprgrad/layers/dnn.nim` modules.\n\n#### Instructions\n\nIn addition to the basic operators `+`, `-`, `*`, `/`, `div`, `mod`, `==`, `\u003c`, `\u003e`, `\u003c=` and `\u003e=`, the following instructions are supported:\n\n| Instruction          | Description                                                       | \n| -------------------- | ----------------------------------------------------------------- |\n| `sq(x)`              | Computes the square of `x`                                        |\n| `min(a, b)`          | Returns the minimum of `a` and `b`                                |\n| `max(a, b)`          | Returns the maximum of `a` and `b`                                |\n| `select(cond, a, b)` | Returns `a` if `cond` is true else returns `b`                    |\n| `sin(x)`             | Returns the sine of `x`                                           |\n| `cos(x)`             | Returns the cosine of `x`                                         |\n| `exp(x)`             | Computes `e ^ x`                                                  |\n| `pow(a, b)`          | Computes `a ^ b`                                                  |\n| `sqrt(x)`            | Computes the square root of `x`                                   |\n| `ln(x)`              | Computes the natural logarithm of `x`                             |\n| `log2(x)`            | Computes the logarithm of base 2 of `x`                           |\n| `log10(x)`           | Computes the logarithm of base 10 of `x`                          |\n| `wrap(x, y)`         | Computes `(x mod y + y) mod y` (`∈ [0, y) ∩ ℤ`)                   |\n| `toScalar(x)`        | Converts `x` to a `Scalar` value                                  |\n| `toIndex(x)`         | Converts `x` to an `Index` value                                  |\n| `tensor.shape[dim]`  | Returns the size of dimension `dim` of `tensor`                   |\n| `tensor.len`         | Returns the number of items in `tensor`                           |\n| `tensor.shape.len`   | Returns the rank of `tensor`                                      |\n| `epoch()`            | Returns the current epoch stored in `Model.epoch`.                |\n| `arr.len`            | Returns the length of the given array.                            |\n| `arr[index]`         | Gets the element stored at `index` in the array.                  |\n\nIf you cannot find the instruction you are looking for, please open an issue.\n\n### Computation Graphs\n\nNeural networks are represented as computation graphs.\nEach computation graph has a set of inputs which are provided to it at run time.\nThey may be images the model is supposed to classify or a text whose sentiment it is supposed to predict.\nEach neural network also has a set of parameters.\nThese are the internal values which are learned during backpropagation.\nExprgrad refers to the output of a given computation as a target.\nA target might be the actual output of the network itself, but also the loss with respect to a training dataset or the action of updating the parameters of the network using gradient descent.\nIn order to compute the value of a target, a series of kernels (layers) is executed.\nAdditionally a computation graph may include a set of caches used to save the internal state of an optimizer and randomized tensors used as inputs to dropout layers.\n\n```nim\nproc param*(shape: openArray[int],\n            initRange: HSlice[float64, float64] = -0.1..0.1,\n            name: string = \"\"): Fun\n```\n\nCreates a new parameter with the given shape.\nEach parameter is randomly initialized with a uniform distribution in the range `initRange` after model compilation.\n\n```nim\nproc input*(name: string, shape: openArray[int] = []): Fun\n```\n\nCreates a new input with the given name.\nThe sizes of static dimensions may be specified to enable compiler optimizations.\nIf a shape is specified unknown dimensions should have the size `-1`.\n\nExample: `input(\"x\", [-1, 28, 28, 1])`\n\n```nim\nproc target*(fun: Fun, name: string): Fun\n```\n\nCreates a new target with the given name.\nTargets may be called using the `Model.call`, `Model.apply` or `Model.fit` procedures.\n\n```nim\nproc backwards*(fun: Fun): Fun\n```\n\nLazily computes the gradients for all parameters of the given computation graph (`fun`) with respect to the given loss value `fun`.\nUnused gradients are not computed.\n\n```nim\nproc optimize*(gradients: Fun,\n               params: HashSet[Fun],\n               optim: proc (param: var Fun, grad: Fun)): Fun\nproc optimize*(gradients: Fun, optim: proc (param: var Fun, grad: Fun)): Fun\n```\n\nOptimizes the given parameters using the given optimizer.\nOptimizers may be created using `makeOpt`.\nThe `Fun.params` procedure may be used to find all parameters of a computation graph.\n\n```nim\nproc backprop*(loss: Fun, optim: proc (param: var Fun, grad: Fun)): Fun\n```\n\nComputes the gradients for all parameters of `loss` and optimizes them using the given optimizer.\nOptimizers may be created using `makeOpt`.\nShortcut for `loss.backwards().optimize(optim)`.\n\n```nim\nproc reshape*(fun: Fun, shape: openArray[int]): Fun\n```\n\nChanges the shape of the given tensor.\nEach reshape may include at most one unknown dimension, which should have the value `-1`.\nThe length of the tensor must stay constant.\n\nExample: `x.reshape([-1, 28 * 28])`\n\n```nim\nproc cond*(branches: openArray[(string, Fun)],\n           otherwise: Fun = nil): Fun\n```\n\nSelects one of the inputs depending on which target should be evaluated.\nUseful for building complex architectures such as GANs.\n\n```nim\nmacro makeOpt*(opt: typed, args: varargs[untyped]): untyped\n```\n\nCreate an optimizer from procedure `opt` by setting all optional arguments of `opt`.\nThe first two arguments to `opt` are the parameter to optimize and its gradient.\nThey must have the types `var Fun` and `Fun`.\n`opt` may not return a value.\n\nExample: `adam.makeOpt(0.01, beta1=0.5)`\n\n### Models\n\n```nim\nproc compile*[T](graphs: varargs[Fun]): Model[T]\n```\n\nCompiles a computation graph to a model.\nThe generic parameter `T` may be one of `float32` or `float64`.\n\n```nim\nproc call*[T](model: Model[T],\n              target: string,\n              args: openArray[(string, Tensor[T])]): Tensor[T]\n```\n\nComputes the value of `target` for the inputs `args`.\n\n```nim\nproc apply*[T](model: Model[T],\n               target: string,\n               args: openArray[(string, Tensor[T])])\n```\n\nComputes `target` and discards its value.\nThis procedure is useful for optimizing simple models.\nIn most cases `Model.fit` should be preferred since it can train in batches and automatically increments `model.epoch`.\n\n```nim\nproc fit*[T](model: Model[T],\n             targetName: string,\n             args: openArray[(string, Tensor[T])],\n             batchSize: int = 32,\n             logStatus: bool = true)\n```\n\nComputes the given target for all batches from the inputs `args`.\nIf the sample count is not divisible by the `batchSize`, the remaining samples are not used in the training process.\nThis will likely be fixed in the future.\n\n```nim\nproc emitIr*[T](model: Model[T]): string\n```\n\nEmits intermediate representation for all targets of `model`.\nThis is mainly used for debugging purposes.\n\n### IO\n\nExprgrad provides an io module which can load commonly used datasets and save/load models to/from disk.\n\n#### Saving and Loading Models\n\nModels can be saved by calling the `save` procedure from `io/serialize`.\n`loadModel` is used to load a model from a file.\nSince `loadModel` loads the intermediate representation for the model from the file and compiles it using the JIT compiler, it is **not** recommended to load models from untrusted sources.\n\n```nim\nlet model = loadModel[float32](\"model.bin\")\nmodel.save(\"model.bin\")\n```\n\n### Tensors\n\nExprgrad currently uses a simple tensor library providing basic functions aimed at preprocessing datasets for training.\nTensors can be created using `Tensor.new` and `Tensor.rand`, printed using `$` and accessed using the `[]` and `{}` operators.\nRefer to `test/test_tensors.nim` for more examples of how to use the tensor library.\n\n## References\n\nExprgrad borrows many successful concepts from other projects on array and differentiable programming languages.\n\n- [Halide](https://halide-lang.org/)\n- [Zygote.jl](https://github.com/FluxML/Zygote.jl)\n- [LLVM](https://llvm.org/)\n\n## Contributing\n\nCurrently exprgrad is still very early in its development.\nAll examples shown above already work, but there are still many possibilities for improvement:\n\n- Improved multithreading\n- GPU Support\n- More automatic optimizations (tiling, loop fusion, ...)\n- ...\n\nIf you would like to contribute to exprgrad, the following tasks might be of interest to you:\n\n- Integrate with existing tensor libraries\n- Image loading and saving\n- Improve batching in `fit` procedure\n- Document the tensors module\n\n### Project Structure\n\nThe following diagram shows a simplified compilation pipeline which displays the functions of the different modules (files in `exprgrad/`) of exprgrad's compiler.\n\n```\n          parser       passes       llvmgen\nNim AST –––––––––\u003e IR ––––––––\u003e IR –––––––––\u003e LLVM IR ––\u003e Machine Code \n```\n\nExprgrad's compiler uses a custom intermediate representation (IR).\nAll program transformations including the automatic differentiation and optimization are performed within this representation.\nIt is defined in the module `ir.nim`.\nThe current compilation pipeline is defined in the `compile` procedure of the module `model.nim`.\nAll program transformations are currently defined in `passes.nim`.\nExprgrad uses the LLVM C-API through its own wrapper.\nThe LLVM IR generator and JIT compiler are defined in `llvmgen.nim`.\n\n## License\n\nCopyright 2021 - 2022 Can Joshua Lehmann\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n  http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n","funding_links":[],"categories":["Nim"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcan-lehmann%2Fexprgrad","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcan-lehmann%2Fexprgrad","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcan-lehmann%2Fexprgrad/lists"}