{"id":30620293,"url":"https://github.com/KarelPeeters/Kyanite","last_synced_at":"2025-08-30T13:10:31.723Z","repository":{"id":196551616,"uuid":"696427406","full_name":"KarelPeeters/Kyanite","owner":"KarelPeeters","description":"A neural network inference library, written in Rust.","archived":false,"fork":false,"pushed_at":"2024-07-13T22:43:11.000Z","size":16456,"stargazers_count":63,"open_issues_count":2,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-08-16T19:49:08.163Z","etag":null,"topics":["inference","machine-learning","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KarelPeeters.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-25T18:11:08.000Z","updated_at":"2025-06-07T10:41:11.000Z","dependencies_parsed_at":"2023-10-14T15:53:09.017Z","dependency_job_id":"1c5a6615-b173-45d2-a164-c53d337c8b91","html_url":"https://github.com/KarelPeeters/Kyanite","commit_stats":null,"previous_names":["karelpeeters/kyanite"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/KarelPeeters/Kyanite","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KarelPeeters%2FKyanite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KarelPeeters%2FKyanite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KarelPeeters%2FKyanite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KarelPeeters%2FKyanite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KarelPeeters","download_url":"https://codeload.github.com/KarelPeeters/Kyanite/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KarelPeeters%2FKyanite/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272853117,"owners_count":25004202,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-30T02:00:09.474Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["inference","machine-learning","rust"],"created_at":"2025-08-30T13:10:25.459Z","updated_at":"2025-08-30T13:10:31.701Z","avatar_url":"https://github.com/KarelPeeters.png","language":"Rust","funding_links":[],"categories":["Neural Networks"],"sub_categories":[],"readme":"# Kyanite\n\n[//]: # (TODO write separate main page doc per each crate)\n[//]: # (TODO   it doesn't need to be a readme, just a lib.rs doc is fine)\n\n[![Crates.io kn-graph](https://img.shields.io/crates/v/kn-graph?label=kn-graph)](https://crates.io/crates/kn-graph)\n[![Crates.io kn-cuda-sys](https://img.shields.io/crates/v/kn-cuda-sys?label=kn-cuda-sys)](https://crates.io/crates/kn-cuda-sys)\n[![Crates.io kn-cuda-eval](https://img.shields.io/crates/v/kn-cuda-eval?label=kn-cuda-eval)](https://crates.io/crates/kn-cuda-eval)\n[![docs.rs](https://img.shields.io/docsrs/kn-graph)](https://docs.rs/releases/search?query=kyanite+kn-)\n![CI status](https://github.com/KarelPeeters/Kyanite/actions/workflows/CI.yml/badge.svg)\n\n\u003c!-- TOC --\u003e\n* [Kyanite](#kyanite)\n  * [Overview](#overview)\n  * [Quick demo](#quick-demo)\n  * [System requirements](#system-requirements)\n  * [Internals](#internals)\n    * [Graph IR](#graph-ir)\n    * [Optimizer](#optimizer)\n    * [CPU executor](#cpu-executor)\n    * [Cuda Executor](#cuda-executor)\n  * [Comparison to other crates](#comparison-to-other-crates)\n    * [Rust wrappers around existing runtimes](#rust-wrappers-around-existing-runtimes)\n    * [From-scratch Rust projects](#from-scratch-rust-projects)\n  * [Development](#development)\n\u003c!-- TOC --\u003e\n\n## Overview\n\nKyanite is a neural network inference library, written in/for Rust. It can run ONNX files either on the CPU or on Nvidia\nGPUs using\ncuda/cudnn/cublas.\n\nIt is general enough to run all kinds of networks, it has been tested with:\n\n* Simple fully connected networks\n* ResNet-based CNNs\n* Large language models like [LLaMA](https://arxiv.org/abs/2302.13971)\n* Image generation models like [Stable Diffusion](https://arxiv.org/abs/2112.10752). For a demo see\n  the `stable_diffusion` example in the `kn-runtime` crate.\n\nThe framework consists of the following crates:\n\n* `kn-graph`: The core crate, containing the intermediate representation and the CPU executor.\n* `kn-cuda-sys`: The Cuda FFI bindings, generated with rust-bindgen.\n* `kn-cuda-eval`: The Cuda executor and planner.\n* `kn-runtime`: A wrapper around the other crates to allow selecting between CPU and GPU execution at runtime.\n* `kn-python`: An experimental python wrapper around the runtime crate, using [PyO3](https://crates.io/crates/pyo3).\n\n## Quick demo\n\n```rust\n// Graph operations (using kn-graph)\n// Load on onnx file into a graph\nlet graph = load_graph_from_onnx_path(\"test.onnx\", false)?;\n// Optimize the graph\nlet graph = optimize_graph(\u0026graph, Default::default());\n// Render the graph as an svg file\ngraph_to_svg(\"test.svg\", \u0026graph, false, false)?;\n\n// Build the inputs\nlet batch_size = 8;\nlet inputs = [DTensor::F32(Tensor::zeros(IxDyn(\u0026[batch_size, 16])))];\n\n// CPU: (using kn-graph)\n// just evaluate the graph\nlet outputs: Vec\u003cDTensor\u003e = cpu_eval_graph(\u0026graph, batch_size, \u0026inputs);\n\n// GPU: (using kn-cuda-eval)\n// build an executor\nlet device = CudaDevice::new(0).unwrap();\nlet mut executor = CudaExecutor::new(device, \u0026graph, batch_size);\n// run the executor on the inputs\nlet outputs: \u0026[DTensor] = executor.evaluate(\u0026inputs);\n\n// Runtime device selection: (using kn-runtime)\nlet device = Device::best();\nlet mut prepared = device.prepare(graph, batch_size);\nlet outputs: Vec\u003cDTensor\u003e = prepared.eval( \u0026 inputs);\n```\n\n## System requirements\n\nTo use the CUDA crates, the appropriate libraries need to be installed on this system; they are not downloaded\nautomatically:\n\n* CUDA (includes CUDA, cuBLAS, NVRTC): [installer](https://developer.nvidia.com/cuda-downloads), follow the\n  instructions.\n  Ensure that the environment variable `CUDA_PATH` points to the root directory of the install (i.e., `CUDA_PATH/bin/`\n  should exist).\n* cuDNN: [archive file](https://developer.nvidia.com/cudnn), to be extracted to a location of your choosing.\n  If you choose the same location as `CUDA_PATH`, you don't need to do anything else. Otherwise, set the environment\n  variable `CUDNN_PATH` to the root directory of the cuDNN installation (i.e., `CUDNN_PATH/bin` should exist).\n\nThe project has been tested with CUDA `v12.2` and cuDNN version `v8.9.5`. Newer versions might work, but this is not\nguaranteed since CUDA sometimes changes the name of or removes certain functions.\n\n## Internals\n\nThe typical pipeline is shown in the first figure below. The second figure shows the results of running this pipeline on\na simple NN architecture.\n\n![NN inference diagram](./docs/arch_inference.svg)\n\n![conv_bn_sm_flow.svg](./docs/conv_bn_sm_flow.svg)\n\n### Graph IR\n\n[//]: # (TODO separate onnx loading chapter, with some examples and tricks)\n\n[//]: # (TODO link/explain supported ONNX subset)\n\n[//]: # (TODO explain strict orthogonality of IR and why it's better than ONNX)\n\nCentral is the _Graph IR_, the intermediate representation for neural network graphs.\n\nThe structure is an [SSA](https://en.wikipedia.org/wiki/Static_single-assignment_form)-style directed acyclic graph,\nwhere nodes are values with a shape, data type, and the operation that computes it. These values are abstract; they\ndon't have strides or memory locations yet.\n\nThe operations are similar to those of other frameworks but are kept as orthogonal as possible. Some example operations:\nconvolution, matmul, reshape, broadcast, slice, unary, binary, reduce, softmax, ...\nSee [the docs](https://docs.rs/kn-graph/latest/kn_graph/graph/enum.Operation.html) for the full list of graph\noperations.\n\nThe graph can be constructed directly in code using\nthe [graph builder API](https://docs.rs/kn-graph/latest/kn_graph/graph/struct.Graph.html), but for convenience, an ONNX\nloader exists. It can read ONNX files and convert the supported subset of operations into those supported by the IR.\n\nBecause the graph IR is much more orthogonal than the ONNX specification, many ONNX operations are decomposed into\nseparate steps, some examples:\n\n* ONNX binary operations implicitly broadcast their operands, but this step is a separate operation in the IR.\n* ONNX convolution and matmul have a built-in optional bias operand; this also becomes a separate broadcast plus binary\n  addition operation.\n\nTo figure out if an ONNX operation is supported, check the branches of the top-level match statement in the `visit_node`\nfunction in [`load.rs`](https://github.com/KarelPeeters/Kyanite/blob/master/kn-graph/src/onnx/load.rs). Many common\noperations are already implemented, and adding more operations shouldn't be too hard.\n\nFor a larger example of a typical graph, see [stable_diffusion_piece.svg](./docs/stable_diffusion_piece.svg), a small\nsection taken from the start of the stable diffusion model.\n\n### Optimizer\n\nThe graph can optionally be optimized by the _optimizer_. Since the graph is append-only, a new graph is returned.\n\nThe optimizations that are currently implemented are:\n\n* Constant folding\n* Fusing consecutive affine (bias, scale, batchnorm) operations into a single bias+scale operation.\n* Fusing consecutive clamping operations (relu, min, max) into a single min+max operation.\n* Strength reduction: replacing division by a constant with multiplication by the inverse constant.\n* Recognizing the layernorm template (reduce, subtract, power, reduce, divide) and replacing it with the layernorm\n  operator.\n\n### CPU executor\n\nFinally, the graph needs to be executed. There is a simple _CPU executor_ that just directly runs each operation. No\nmajor optimizations are attempted here, except for using BLAS routines for matmuls and im2col for convolutions. It's\nimportant that this executor is as simple as possible because it serves as the baseline for unit tests that check the\ncorrectness of the GPU executor.\n\n### Cuda Executor\n\nThe second (and more useful) way to run these graphs is with the _Cuda executor_. This involves running the graph\nthrough the _Cuda Planner_, which outputs a predetermined schedule of Cuda operations and allocates the necessary memory\nbuffers. This is split out as a separate step so this expensive planning step only needs to be carried out once per\nnetwork architecture; the resulting plan can then be reused many times in the executor.\n\nThe planner has the following major responsibilities:\n\n* Determine the memory layout of tensors: the strides and the memory offsets\n    * This implicitly handles most reshape, broadcast, stride, ... operations.\n    * Buffers are also reused if possible, minimizing total memory usage. There is much room for improvement here;\n      currently, this is just a single pass algorithm.\n\n[//]: # (TODO show memory usage graphs?)\n\n* Decide which cuDNN/cuBLAS operations to run for convolutions and matmuls. If possible, operations are fused together.\n  Some examples:\n    * cuDNN supports a single \"convolution + residual + bias + relu\" operation\n    * cuBLAS matmuls can include a transpose of either input matrix, and equivalently the output by swapping the inputs.\n    * cuDNN and cuBLAS operations sometimes include a \"scalar\" argument that is multiplied by some of the operands\n\n[//]: # (TODO more fusing examples: cuBLAS + scale + transpose, are there others?)\n\n* Compile custom kernels for the remaining scalar and compound operations using an _autokernel_ framework based\n  on [NVRTC (Runtime Compilation)](https://docs.nvidia.com/cuda/nvrtc/index.html).\n    * The operations handled by *autokernel* are: scalar operations, reduce, softmax, layernorm, gather.\n    * Handwritten kernel templates are used, with details such as tensor shapes, strides, scalar operations, ...\n      substituted in before compilation at runtime.\n    * More operator fusion happens here\n        * Multiple scalar operations get compiled to a single kernel\n        * Constant scalars are inlined\n        * Some compound kernels support fusing input or output scalar operations\n\nThis final operator fusion can be significant and save a lot of redundant transfers to and from main memory. The same\nperformance could be achieved by manually writing kernels for each used combination of operations, but the combinatorial\nexplosion and associated maintenance would be huge.\n\nAn example generated scalar kernel with some handwritten clarifying comments is shown below:\n\n\u003cdetails\u003e\n\u003csummary\u003eExample scalar autokernel for residual + batchnorm + relu6\u003c/summary\u003e\n\n```cpp\n#include \"util.cu\"\n\n// constants that got inserted into the template\n// this scalar operation happens on a tensor of rank 4, with 7 operands\nconst int RANK = 4;\nconst int OPERANDS = 7;\nconst int STRIDES_DENSE[RANK] = {648, 81, 9, 1};\nconst int STRIDES[OPERANDS][RANK] = {\n    // these are full input tensors with normal, dense strides\n    {648, 81, 9, 1},\n    {648, 81, 9, 1},\n    // these values have zero strides for all axes except the channel one,\n    //    so these are probably biases and scaling factors\n    //    that are broadcast across the other axes\n    {0, 1, 0, 0},\n    {0, 1, 0, 0},\n    {0, 1, 0, 0},\n    {0, 1, 0, 0},\n    // the output tensor is just another operand\n    {648, 81, 9, 1}\n};\n\n// the template function, the body of which is generated at runtime\n__device__ void operation(void *pointers[OPERANDS], int offsets[OPERANDS]) {\n    // all input operand memory locations are cast to the right type\n    float *x0 = \u0026((float *) pointers[0])[offsets[0]];\n    float *x1 = \u0026((float *) pointers[1])[offsets[1]];\n    float *x2 = \u0026((float *) pointers[2])[offsets[2]];\n    float *x3 = \u0026((float *) pointers[3])[offsets[3]];\n    float *x4 = \u0026((float *) pointers[4])[offsets[4]];\n    float *x5 = \u0026((float *) pointers[5])[offsets[5]];\n    float *x6 = \u0026((float *) pointers[6])[offsets[6]];\n    \n    // input operands are loaded\n    float y0 = *x0;\n    float y1 = *x1;\n    \n    // this is probably a residual connection\n    float y2 = y0 + y1;\n    \n    // these 4 steps look like they're implementing a batchnorm layer  \n    float y3 = *x2;\n    float y4 = y2 - y3;\n    float y5 = *x3;\n    float y6 = y4 / y5;\n    float y7 = *x4;\n    float y8 = y6 * y7;\n    float y9 = *x5;\n    float y10 = y8 + y9;\n    \n    // this implements a relu6 activation function\n    float y11 = 6;\n    float y12 = min(y10, y11);\n    float y13 = (0.0);\n    float y14 = max(y12, y13);\n    \n    // finally the output is stored\n    *x6 = y14;\n}\n\n// the kernel main function is the same for all scalar kernels\n__global__ void scalar_kernel(\n        int batch_size,\n        Array\u003cvoid *, OPERANDS\u003e pointers\n) {\n    KernelInfo info = kernel_info();\n    int size = batch_size * STRIDES_DENSE[0];\n\n    // the main loop, following https://developer.nvidia.com/blog/cuda-pro-tip-write-flexible-kernels-grid-stride-loops/\n    for (int flat = info.global_thread_id; flat \u003c size; flat += info.thread_count) {\n        Array\u003cint, OPERANDS\u003e offsets = flat_index_to_offsets\u003cRANK, OPERANDS\u003e(flat, STRIDES_DENSE, STRIDES);\n        operation(pointers.data, \u0026offsets[0]);\n    }\n}\n```\n\u003c/details\u003e\n\n## Comparison to other crates\n\nSee [Are We Learning Yet?](https://www.arewelearningyet.com/neural-networks/) for a full list of potential alternatives.\n\n### Rust wrappers around existing runtimes\n\n* PyTorch wrapper: [tch](https://crates.io/crates/tch)\n* TensorFlow wrapper: [tensorflow](https://crates.io/crates/tensorflow)\n* ONNXRuntime wrapper: [ort](https://github.com/pykeio/ort)\n\nPositives:\n* extensive support for many neural network operations\n* support for many different backends (CPU, GPU (Nvidia + AMD), TPU, ...)\n\nNegatives\n* not always great support for loading ONNX files (ort is great at this though, as the name suggests)\n* large and somewhat black-box external dependency\n* less operator fusion in many cases, although this is expected to improve in the future\n\nPerformance should be about the same as Kyanite for cases where operator fusion does not matter much; all libraries\nmostly use the same underlying cuDNN and cuBLAS kernels.\n\n### From-scratch Rust projects\n\n* [tract](https://github.com/sonos/tract): larger coverage of the ONNX specification but only does CPU inference\n\n## Development\n\nWhile developing this crate, to update the ONNX proto, the [prost-build crate](https://crates.io/crates/prost-build) is\nused. This requires that `protoc` is installed and that the `PROTOC` environment variable is set to point to the\nexecutable. See their installation instructions (or the error message the build script shows if any) for more details.\n\nTo actually update the proto definition, replace `kn-graph/proto/onnx.proto3` with the newer version and\nrun `cargo run --bin proto-to-rust`. Then commit both the `onnx.proto3` file and the generated `onnx.rs` file.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FKarelPeeters%2FKyanite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FKarelPeeters%2FKyanite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FKarelPeeters%2FKyanite/lists"}