{"id":13604454,"url":"https://github.com/MachineLearningSystem/DVABatch","last_synced_at":"2025-04-12T02:30:36.106Z","repository":{"id":185461741,"uuid":"513441314","full_name":"MachineLearningSystem/DVABatch","owner":"MachineLearningSystem","description":null,"archived":false,"fork":true,"pushed_at":"2022-05-13T01:41:35.000Z","size":50,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2024-11-07T09:42:40.134Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"sjtu-epcc/DVABatch","license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MachineLearningSystem.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-07-13T08:31:21.000Z","updated_at":"2022-06-22T03:20:35.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/MachineLearningSystem/DVABatch","commit_stats":null,"previous_names":["machinelearningsystem/dvabatch"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MachineLearningSystem%2FDVABatch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MachineLearningSystem%2FDVABatch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MachineLearningSystem%2FDVABatch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MachineLearningSystem%2FDVABatch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MachineLearningSystem","download_url":"https://codeload.github.com/MachineLearningSystem/DVABatch/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248506901,"owners_count":21115503,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T19:00:45.991Z","updated_at":"2025-04-12T02:30:35.828Z","avatar_url":"https://github.com/MachineLearningSystem.png","language":null,"readme":"# DVABatch\n\nThis repository contains the source code for a research paper that was submitted for publication at the [2022 USENIX Annual Technical Conference](https://www.usenix.org/conference/atc22) (ATC22).\n\n## What is DVABatch\n\nThe DNN inferences are often batched for better utilizing the hardware in existing DNN serving systems. However, DNN serving exhibits diversity in many aspects, such as input, operator, and load. The unawareness of these diversities results in inefficient processing. Our investigation shows that the inefficiency roots in the feature of existing batching mechanism: one entry and one exit. Therefore, we propose **DVABatch**, a runtime batching system that enables the multi-entry multi- exit batching scheme for existing DNN serving system.\n\n## Environment Preparation\n\n- Hardware\u0026software requirements\n\n  1. Hardware Requirements\n\n     1. CPU: Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz\n     2. Memroy: 252G\n     3. NVIDIA TitanRTX\n\n  2. Software Requirements\n\n     1. Ubuntu 18.04.6 (Kernel 4.15.0)\n     2. GPU Driver: 460.39\n     3. CUDA 11.3\n     4. CUDNN 8.2\n     5. TensorRT 8.0.3.4\n     6. RapidJSON\n     7. cmake 3.17\n\n- Some software installation tips\n  \n  1. Environment variables should be added for TensorRT and RapidJSON, including `PATH, LIBRARY_PATH,  LD_LIBRARY_PATH, CMAKE_PREFIX_PATH, CPLUS_INCLUDE_PATH`.\n\n## Getting Start\n\n- Assuming you have the above requirements and you are in the `$HOME` directory, you can clone the repository and start the installation.\n\n```bash\nmkdir $HOME/DVABatch\ngit clone git@github.com:sjtu-epcc/DVABatch.git DVAbatch/lego\n```\n\n- We use following instructions to compile the DVABatch runtime system.\n\n```shell\ncd $HOME/DVABatch/lego\nmkdir build\ncd build\ncmake -DCMAKE_INSTALL_PREFIX=~/DVABatch/install ..\nmake -j\nmake install\n```\n\n- This repo only consists the source files for DVABatch runtime system. DVABatch relies on [Triton Inference Server](https://github.com/triton-inference-server/server.git) to provide DNN services. Therefore, we need to install Triton Inference Server. We also provide a customized Triton Inference Server [here](https://github.com/Raphael-Hao/lego_server) for DVABatch in this repo for simple test. We now build the customized Triton Inference Server.\n\n```bash\ngit clone https://github.com/Raphael-Hao/lego_server.git $HOME/DVABatch/server\ncmake -DCMAKE_INSTALL_PREFIX=~/DVABatch/install ../build\nmake server\n```\n\n- User needs to prepare the sliced models and then we can simply run DVABatch with following commnads.\n\n```bash\ncd $HOME/DVABatch/install\n./bin/benchmark ../server/config/resnet_152_04.json\n```\n","funding_links":[],"categories":["Paper-Code"],"sub_categories":["Serving-Inference"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMachineLearningSystem%2FDVABatch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMachineLearningSystem%2FDVABatch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMachineLearningSystem%2FDVABatch/lists"}