{"id":21168486,"url":"https://github.com/jetbrains-research/kinference","last_synced_at":"2025-04-04T16:13:09.912Z","repository":{"id":38365751,"uuid":"244400016","full_name":"JetBrains-Research/kinference","owner":"JetBrains-Research","description":"Running ONNX models in vanilla Kotlin","archived":false,"fork":false,"pushed_at":"2024-11-29T13:49:43.000Z","size":62583,"stargazers_count":169,"open_issues_count":8,"forks_count":7,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-03-28T15:05:48.181Z","etag":null,"topics":["kotlin","machine-learning","neural-networks","onnx"],"latest_commit_sha":null,"homepage":"","language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JetBrains-Research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-02T15:04:54.000Z","updated_at":"2025-03-27T14:20:00.000Z","dependencies_parsed_at":"2024-08-29T12:57:44.863Z","dependency_job_id":"611ce767-8043-4348-b2f9-44baf994f073","html_url":"https://github.com/JetBrains-Research/kinference","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JetBrains-Research%2Fkinference","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JetBrains-Research%2Fkinference/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JetBrains-Research%2Fkinference/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JetBrains-Research%2Fkinference/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JetBrains-Research","download_url":"https://codeload.github.com/JetBrains-Research/kinference/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247208139,"owners_count":20901570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kotlin","machine-learning","neural-networks","onnx"],"created_at":"2024-11-20T15:14:09.052Z","updated_at":"2025-04-04T16:13:09.892Z","avatar_url":"https://github.com/JetBrains-Research.png","language":"Kotlin","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1\u003e \u003cimg align=\"center\" width=\"64\" height=\"64\" src=\"https://s3-eu-west-1.amazonaws.com/public-resources.ml-labs.aws.intellij.net/static/kinference/icon_256.png\" alt=\"KInference Icon\"\u003e KInference \u003c/h1\u003e\n\n[![JB Research](https://jb.gg/badges/research-flat-square.svg)](https://research.jetbrains.org/)\n\nKInference is a library that makes it possible to execute complex ML models (written via ONNX) in Kotlin.\n\n[ONNX](https://github.com/onnx/onnx) is a popular ecosystem for building, training, evaluating, and exchanging ML and DL models. It makes the process much\nsimpler and divides the model into building blocks that can be switched or tuned to one's liking.\n\nHowever, popular ML libraries, including those intended for the inference of ONNX models, carry with themselves \na lot of dependencies and requirements that complicate their use in some cases. \nKInference is designed to facilitate the inference of ONNX models on a variety of platforms via configurable backends.\nOur library addresses not only the problem of server side inference, but also of local inference as well, and provides \nseveral solutions that are suitable for running both on user side and server side. \n\nRight now, KInference is in active development.\n\n### Table of contents:\n* [Why should I use KInference?](#why-should-i-use-kinference)\n* [KInference Backends](#kinference-backends)\n* [Third-party math libraries adapters](#third-party-math-libraries-adapters)\n* [Getting started](#getting-started)\n* [Examples](#examples)\n\n### Why should I use KInference?\n\n* **KInference is specifically optimized for inference.**\n  Most of the existing ML libraries are, in fact, versatile tools for model training and inference, \n  but carry with themselves a lot of dependencies and requirements. KInference, on the other hand, addresses inference-only functionality\n  to help facilitate model inference with a relatively small yet convenient API and inference-specific optimizations.\n\n* **KInference has pure-JS and pure-JVM backends** that make it possible to run models anywhere where JS or JVM virtual machine is available.\n  In addition, you can switch between the chosen backends using backend configuration.\n\n* **KInference supports configurable backends.**\n  KInference employs platform-specific optimizations and allows backend configuration essential for multiplatform projects.\n  You can choose a backend for every module in the `build.gradle.kts` project file just by adding corresponding dependencies, \n  while keeping most of your KInference-related code in a single common module.\n\n* **KInference enables data preprocessing.** \n  We understand that data needs preprocessing before feeding it to the model and that is why we implemented numpy-like n-dimensional arrays.\n  KInference can also work with custom array formats, with some of them being available out-of-the-box\n  (see [multik](https://github.com/Kotlin/multik), \n  [kmath](https://github.com/SciProgCentre/kmath)).\n\n## KInference backends\n\n### KInference Core\nPure Kotlin implementation that requires nothing but vanilla Kotlin. KInference Core is lightweight but fast, and supports numerous ONNX operators.\nIt makes the library easy to use and especially convenient for various applications that require the models to run locally on users' machines.\nNote that this backend is well-optimized for JVM projects only, and, despite the fact that KInference Core is available for JavaScript projects, \nit is highly recommended to use KInference TensorFlow.js backend instead for more performance.\n\nKInference Core dependency coordinates:\n```kotlin\ndependencies {\n    api(\"io.kinference\", \"inference-core\", \"0.2.26\")\n}\n```\n\n### TensorFlow.js\nHigh-performance JavaScript backend that relies on the [Tensorflow.js](https://www.tensorflow.org/js/) library. \nEssentially, it employs GPU operations provided by TensorFlow.js to boost the computations. \nIn addition, this implementation enables model execution directly in the user's browser.\nThis backend is recommended for JavaScript projects.\n\nTensorFlow.js backend dependency coordinates:\n```kotlin\ndependencies {\n    api(\"io.kinference\", \"inference-tfjs\", \"0.2.26\")\n}\n```\n\n### ONNXRuntime CPU and ONNXRuntime GPU\nJava backends that use [ONNXRuntime](https://github.com/microsoft/onnxruntime) as an inference engine \nand provide common KInference API to interact with the ONNXRuntime library.\n\nNote that the GPU backend is **CUDA-only**.\nTo check on the system requirements, visit the following [link](https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements)\n\nONNXRuntime CPU backend dependency coordinates:\n```kotlin\ndependencies {\n    api(\"io.kinference\", \"inference-ort\", \"0.2.26\")\n}\n```\n\nONNXRuntime GPU backend dependency coordinates:\n```kotlin\ndependencies {\n    api(\"io.kinference\", \"inference-ort-gpu\", \"0.2.26\")\n}\n```\n\n## Third-party math libraries adapters\nKInference works with custom array formats, and some of them are available out-of-the-box.\nBasically, adapters enable working with familiar array formats and libraries. \nYou can use several third-party Kotlin math libraries with KInference via our data adapters.\nIn addition to the library adapters listed below, you can implement your own adapters using KInference adapters API.\n\n### KMath adapter\nArray adapter for the [kmath](https://github.com/SciProgCentre/kmath) library that works with JVM KInference backends.\n\nDependency coordinates:\n```kotlin\ndependencies {\n    api(\"io.kinference\", \"adapter-kmath-{backend_name}\", \"0.2.26\")\n}\n```\n\n### Multik adapter\nArray adapter for the [multik](https://github.com/Kotlin/multik) library that works with JVM KInference backends.\n\nDependency coordinates:\n```kotlin\ndependencies {\n    api(\"io.kinference\", \"adapter-multik-{backend_name}\", \"0.2.26\")\n}\n```\n\n## Getting started\nLet us now walk through how to get started with KInference. The latest version of KInference is *0.2.26*\n\n### Setup dependencies repository\n\nFirstly, you should add KInference repository in `build.gradle.kts` via:\n\n```kotlin\nrepositories {\n    maven {\n        url = uri(\"https://packages.jetbrains.team/maven/p/ki/maven\")\n    }\n    \n    maven {\n        url = uri(\"https://packages.jetbrains.team/maven/p/grazi/grazie-platform-public\")\n    }\n}\n```\n\n### Project setup\nTo enable the backend, you can add the chosen KInference runtime as a dependency:\n\n```kotlin\ndependencies {\n    api(\"io.kinference\", \"inference-core\", \"0.2.26\")\n}\n```\n\n### Multi-backend project setup\nTo configure individual KInference backend for each target, you should add corresponding backends to the dependencies.\n\n```kotlin\nkotlin {\n    jvm {}\n\n    js(IR) {\n        browser()\n    }\n\n    sourceSets {\n        val commonMain by getting {\n            dependencies {\n                api(\"io.kinference:inference-api:0.2.26\")\n                api(\"io.kinference:ndarray-api:0.2.26\")\n            }\n        }\n\n        val jvmMain by getting {\n            dependencies {\n                api(\"io.kinference:inference-core:0.2.26\")\n            }\n        }\n\n        val jsMain by getting {\n            dependencies {\n                api(\"io.kinference:inference-tfjs:0.2.26\")\n            }\n        }\n    }\n}\n```\n\n## Examples\nThe [examples module](https://github.com/JetBrains-Research/kinference/tree/master/examples) contains examples of solving classification tasks\n(cats vs dogs) and text generation. \nDifferent backends are used in the examples.\nModels for the examples were selected from the [ONNX Model Zoo](https://github.com/onnx/models).\nRunning the examples does not require converting models to different opsets.\nHowever, if you need to run a model with operator versions not supported by KInference,\nyou can refer to [Convert guide](https://github.com/OpenPPL/ppl.nn/blob/master/docs/en/onnx-model-opset-convert-guide.md).\n\n## Want to know more?\nKInference API itself is widely documented, so you can explore its code and interfaces to get to know KInference better.\n\nYou may also submit feedback and ask questions in repository issues and issue discussions.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjetbrains-research%2Fkinference","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjetbrains-research%2Fkinference","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjetbrains-research%2Fkinference/lists"}