{"id":13995274,"url":"https://github.com/vertexclique/orkhon","last_synced_at":"2025-04-07T16:18:30.106Z","repository":{"id":50503135,"uuid":"187414091","full_name":"vertexclique/orkhon","owner":"vertexclique","description":"Orkhon: ML Inference Framework and Server Runtime","archived":false,"fork":false,"pushed_at":"2021-02-01T15:23:47.000Z","size":27489,"stargazers_count":149,"open_issues_count":3,"forks_count":4,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-31T14:15:08.257Z","etag":null,"topics":["async","data-parallelism","inference-server","machine-learning","multiprocessing","python3","tensorflow"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vertexclique.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":"vertexclique","patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2019-05-18T23:24:53.000Z","updated_at":"2025-03-23T10:31:58.000Z","dependencies_parsed_at":"2022-08-02T16:15:42.939Z","dependency_job_id":null,"html_url":"https://github.com/vertexclique/orkhon","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vertexclique%2Forkhon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vertexclique%2Forkhon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vertexclique%2Forkhon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vertexclique%2Forkhon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vertexclique","download_url":"https://codeload.github.com/vertexclique/orkhon/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247685628,"owners_count":20979085,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","data-parallelism","inference-server","machine-learning","multiprocessing","python3","tensorflow"],"created_at":"2024-08-09T14:03:19.968Z","updated_at":"2025-04-07T16:18:30.081Z","avatar_url":"https://github.com/vertexclique.png","language":"Rust","funding_links":["https://github.com/sponsors/vertexclique"],"categories":["Rust","Model Inference"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/vertexclique/orkhon/blob/master/doc/logo/orkhon.png\"\u003e\u003cbr\u003e\n\u003c/div\u003e\n\n-----------------\n\n\u003ch1 align=\"center\"\u003eOrkhon: ML Inference Framework and Server Runtime\u003c/h1\u003e\n\n\n\u003ctable align=left style='float: left; margin: 4px 10px 0px 0px; border: 1px solid #000000;'\u003e\n\u003ctr\u003e\n  \u003ctd\u003eLatest Release\u003c/td\u003e\n  \u003ctd\u003e\n    \u003ca href=\"https://crates.io/crates/orkhon\"\u003e\n    \u003cimg alt=\"Crates.io\" src=\"https://img.shields.io/crates/v/orkhon.svg?style=popout-square\"\u003e\n    \u003c/a\u003e\n  \u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd\u003e\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd\u003eLicense\u003c/td\u003e\n  \u003ctd\u003e\n    \u003ca href=\"https://github.com/vertexclique/orkhon/blob/master/LICENSE\"\u003e\n    \u003cimg alt=\"Crates.io\" src=\"https://img.shields.io/crates/l/orkhon.svg?style=popout-square\"\u003e\n    \u003c/a\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd\u003eBuild Status\u003c/td\u003e\n  \u003ctd\u003e\n    \u003ca href=\"https://github.com/vertexclique/orkhon/actions\"\u003e\n    \u003cimg alt=\"Build Status\" src=\"https://github.com/vertexclique/orkhon/workflows/CI/badge.svg\" /\u003e\n    \u003c/a\u003e\n  \u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n  \u003ctd\u003eDownloads\u003c/td\u003e\n  \u003ctd\u003e\n    \u003ca href=\"https://crates.io/crates/orkhon\"\u003e\n    \u003cimg alt=\"Crates.io\" src=\"https://img.shields.io/crates/d/orkhon.svg?style=popout-square\"\u003e\n    \u003c/a\u003e\n  \u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\t\u003ctd\u003eGitter\u003c/td\u003e\n\t\u003ctd\u003e\n\t\t\u003ca href=\"https://gitter.im/orkhonml/community\"\u003e\n\t\t\u003cimg src=\"https://badges.gitter.im/Join%20Chat.svg\" /\u003e\n\t\t\u003c/a\u003e\n\t\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n## What is it?\n\nOrkhon is Rust framework for Machine Learning to run/use inference/prediction code written in Python, frozen models and process unseen data. It is mainly focused on serving models and processing unseen data in a performant manner. Instead of using Python directly and having scalability problems for servers this framework tries to solve them with built-in async API.\n\n## Main features\n\n* Sync \u0026 Async API for models.\n* Easily embeddable engine for well-known Rust web frameworks.\n* API contract for interacting with Python code.\n* High processing throughput\n    * ~4.8361 GiB/s prediction throughput\n    * 3_000 concurrent requests takes ~4ms on average \n* Python Module caching\n\n## Installation\n\nYou can include Orkhon into your project with;\n\n```toml\n[dependencies]\norkhon = \"0.2\"\n```\n\n## Dependencies\nYou will need:\n* If you use `pymodel` feature, Python dev dependencies should be installed and have proper python runtime to use Orkhon with your project.\n* If you want to have tensorflow inference. Installing tensorflow as library for linking is required.\n* ONNX interface doesn't need extra dependencies from the system side.\n* Point out your `PYTHONHOME` environment variable to your Python installation.\n## Python API contract\n\nFor Python API contract you can take a look at the [Project Documentation](https://docs.rs/orkhon).\n\n ## Examples\n #### Request a Tensorflow prediction asynchronously\n\n```rust\n use orkhon::prelude::*;\n use orkhon::tcore::prelude::*;\n use orkhon::ttensor::prelude::*;\n use rand::*;\n use std::path::PathBuf;\n\nlet o = Orkhon::new()\n    .config(\n        OrkhonConfig::new()\n            .with_input_fact_shape(InferenceFact::dt_shape(f32::datum_type(), tvec![10, 100])),\n    )\n    .tensorflow(\n        \"model_which_will_be_tested\",\n        PathBuf::from(\"tests/protobuf/manual_input_infer/my_model.pb\"),\n    )\n    .shareable();\n\nlet mut rng = thread_rng();\nlet vals: Vec\u003c_\u003e = (0..1000).map(|_| rng.gen::\u003cf32\u003e()).collect();\nlet input = tract_ndarray::arr1(\u0026vals).into_shape((10, 100)).unwrap();\n\nlet o = o.get();\nlet handle = async move {\n    let processor = o.tensorflow_request_async(\n       \"model_which_will_be_tested\",\n       ORequest::with_body(TFRequest::new().body(input.into())),\n    );\n    processor.await\n};\nlet resp = block_on(handle).unwrap();\n```\n\n #### Request an ONNX prediction synchronously\n\nThis example needs `onnxmodel` feature enabled.\n\n```rust\nuse orkhon::prelude::*;\nuse orkhon::tcore::prelude::*;\nuse orkhon::ttensor::prelude::*;\nuse rand::*;\nuse std::path::PathBuf;\n\n let o = Orkhon::new()\n     .config(\n         OrkhonConfig::new()\n             .with_input_fact_shape(InferenceFact::dt_shape(f32::datum_type(), tvec![10, 100])),\n     )\n     .onnx(\n         \"model_which_will_be_tested\",\n         PathBuf::from(\"tests/protobuf/onnx_model/example.onnx\"),\n     )\n     .build();\n\n let mut rng = thread_rng();\n let vals: Vec\u003c_\u003e = (0..1000).map(|_| rng.gen::\u003cf32\u003e()).collect();\n let input = tract_ndarray::arr1(\u0026vals).into_shape((10, 100)).unwrap();\n\n let resp = o\n     .onnx_request(\n         \"model_which_will_be_tested\",\n         ORequest::with_body(ONNXRequest::new().body(input.into())),\n     )\n     .unwrap();\n assert_eq!(resp.body.output.len(), 1);\n```\n\n## License\n\nLicense is [MIT](https://github.com/vertexclique/orkhon/blob/master/LICENSE)\n\n## Documentation\n\nOfficial documentation is hosted on [docs.rs](https://docs.rs/orkhon).\n\n## Getting Help\nPlease head to our [Gitter](https://gitter.im/orkhonml/community) or use [StackOverflow](https://stackoverflow.com/questions/tagged/orkhon)\n\n## Discussion and Development\nWe use [Gitter](https://gitter.im/orkhonml/community) for development discussions. Also please don't hesitate to open issues on GitHub ask for features, report bugs, comment on design and more!\nMore interaction and more ideas are better!\n\n## Contributing to Orkhon [![Open Source Helpers](https://www.codetriage.com/vertexclique/orkhon/badges/users.svg)](https://www.codetriage.com/vertexclique/orkhon)\n\nAll contributions, bug reports, bug fixes, documentation improvements, enhancements and ideas are welcome.\n\nA detailed overview on how to contribute can be found in the  [CONTRIBUTING guide](.github/CONTRIBUTING.md) on GitHub.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvertexclique%2Forkhon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvertexclique%2Forkhon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvertexclique%2Forkhon/lists"}