{"id":15828369,"url":"https://github.com/mlomb/ort-batcher","last_synced_at":"2025-03-15T04:31:31.299Z","repository":{"id":207711363,"uuid":"719898365","full_name":"mlomb/ort-batcher","owner":"mlomb","description":null,"archived":false,"fork":false,"pushed_at":"2024-01-22T01:29:05.000Z","size":4713,"stargazers_count":2,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-04-23T14:43:21.495Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mlomb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-17T06:14:16.000Z","updated_at":"2024-04-23T14:43:21.496Z","dependencies_parsed_at":"2024-01-22T02:39:04.544Z","dependency_job_id":"3730a296-dc96-426a-b804-53ced30024bd","html_url":"https://github.com/mlomb/ort-batcher","commit_stats":null,"previous_names":["mlomb/ort-batcher"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlomb%2Fort-batcher","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlomb%2Fort-batcher/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlomb%2Fort-batcher/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlomb%2Fort-batcher/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mlomb","download_url":"https://codeload.github.com/mlomb/ort-batcher/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243685506,"owners_count":20330980,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-05T10:40:23.812Z","updated_at":"2025-03-15T04:31:30.612Z","avatar_url":"https://github.com/mlomb.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ort-batcher\n\n\u003ca href=\"https://crates.io/crates/ort_batcher\" target=\"_blank\"\u003e\n    \u003cimg alt=\"Crates.io\" src=\"https://img.shields.io/crates/v/ort_batcher?style=for-the-badge\u0026logo=rust\"\u003e\n\u003c/a\u003e\n\n\nSmall crate to batch inferences of ONNX models using [ort](https://github.com/pykeio/ort). Inspired by [batched_fn](https://docs.rs/batched-fn/latest/batched_fn/).\n\nNote that it only works with models that:\n* Have their first dimension dynamic (-1), so they can be batched.\n* Inputs and outputs are tensors of type `float32`.\n\n# Usage\n\n```rust\nlet max_batch_size = 32;\nlet max_wait_time = Duration::from_millis(80);\nlet batcher = Batcher::spawn(session, max_batch_size, max_wait_time);\n\n// in some thread\nlet inputs = vec![ArrayD::\u003cf32\u003e::zeros(vec![7, 8, 9])];\nlet outputs = batcher.run(inputs).unwrap();\n```\n\n# Example\n\nCheck [example.rs](examples/example.rs):\n\n```rust\nuse ndarray::{ArrayD, Axis};\nuse ort::{CUDAExecutionProvider, Environment, SessionBuilder, Value};\nuse ort_batcher::batcher::Batcher;\nuse std::time::Duration;\n\nfn main() -\u003e ort::Result\u003c()\u003e {\n    tracing_subscriber::fmt::init();\n\n    ort::init()\n        .with_execution_providers([CUDAExecutionProvider::default().build()])\n        .commit()?;\n\n    let session = Session::builder()?\n        .with_intra_threads(1)?\n        .with_model_from_memory(include_bytes!(\"../tests/model.onnx\"))?;\n\n    {\n        let start = std::time::Instant::now();\n\n        // 128 threads\n        // 256 inferences each\n        // sequential\n        std::thread::scope(|s| {\n            for _ in 0..128 {\n                let session = \u0026session;\n                let input = ArrayD::\u003cf32\u003e::zeros(vec![7, 8, 9]);\n\n                s.spawn(move || {\n                    for _ in 0..256 {\n                        let value = Value::from_array(input.clone().insert_axis(Axis(0))).unwrap();\n                        let _output = session.run([value]).unwrap()[0]\n                            .extract_tensor::\u003cf32\u003e()\n                            .unwrap()\n                            .view()\n                            .index_axis(Axis(0), 0)\n                            .to_owned();\n                    }\n                });\n            }\n        });\n\n        println!(\"sequential: {:?}\", start.elapsed());\n    }\n\n    let max_batch_size = 32;\n    let max_wait_time = Duration::from_millis(10);\n    let batcher = Batcher::spawn(session, max_batch_size, max_wait_time);\n\n    {\n        let start = std::time::Instant::now();\n\n        // 128 threads\n        // 256 inferences each\n        // batched\n        std::thread::scope(|s| {\n            for _ in 0..128 {\n                let batcher = \u0026batcher;\n                let input = ArrayD::\u003cf32\u003e::zeros(vec![7, 8, 9]);\n\n                s.spawn(move || {\n                    for _ in 0..256 {\n                        let _output = batcher.run(vec![input.clone()]).unwrap();\n                    }\n                });\n            }\n        });\n\n        println!(\"batched: {:?}\", start.elapsed());\n    }\n\n    Ok(())\n}\n```\n\nNote that to have good results you have to use heavy model in a GPU, otherwise you may not see any difference.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlomb%2Fort-batcher","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmlomb%2Fort-batcher","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlomb%2Fort-batcher/lists"}