{"id":20768174,"url":"https://github.com/softwaremill/triton_playground","last_synced_at":"2025-03-11T19:28:40.933Z","repository":{"id":156118325,"uuid":"632303945","full_name":"softwaremill/triton_playground","owner":"softwaremill","description":"Triton Inference Server playground with different features to play around.","archived":false,"fork":false,"pushed_at":"2023-05-08T12:56:16.000Z","size":18,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-01-18T06:42:44.230Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/softwaremill.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-25T06:17:21.000Z","updated_at":"2023-04-26T09:24:46.000Z","dependencies_parsed_at":null,"dependency_job_id":"cc08de83-4567-46a2-a0bb-812c4d672f67","html_url":"https://github.com/softwaremill/triton_playground","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softwaremill%2Ftriton_playground","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softwaremill%2Ftriton_playground/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softwaremill%2Ftriton_playground/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softwaremill%2Ftriton_playground/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/softwaremill","download_url":"https://codeload.github.com/softwaremill/triton_playground/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243098023,"owners_count":20235941,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T11:36:14.638Z","updated_at":"2025-03-11T19:28:40.914Z","avatar_url":"https://github.com/softwaremill.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Environment\n\nThe recommended way to setup environemnt is to create Python\nvirtual environment.\n```bash\nvirtualenv -p python3.10 .venv\n```\n\n```bash\nsource .venv/bin/activate\npip install --upgrade pip\npip install -r requirements.txt\n```\n\n# Prepare model-repository\n\nFirst, download the model from `torchhub` and save it as TorchScript and ONNX.\n```bash\npython3 utils/export.py\n```\n\nNext, activate docker conatiner and convert ONNX model to TensorRT.\n```bash\ndocker run --gpus all -it --rm -v $(pwd):/workspace nvcr.io/nvidia/tensorrt:22.04-py3\nbash utils/convert_to_tensorrt.sh\nexit\n```\n\nMove exported models into `model-repository`.\n```bash\nbash utils/move_models.sh\n```\n\n# Run Triton\n\nActivate docker container with Triton server with mounted `model_repository` directory.\n\n```bash\ndocker run -it --gpus all -v /var/run/docker.sock:/var/run/docker.sock -v $(pwd)/model_repository:/models --shm-size 1024m --net=host nvcr.io/nvidia/tritonserver:22.04-py3\n```\n\nNext, install dependencies and run Triton:\n\n```bash\npip install pillow torch transformers\ntritonserver --model-repository=/models\n```\n\n# perf_analzyer\n\nWith the Triton running in another container run the command below to enter\nappropriate container and run `perf_analyzer`:\n\n```bash\ndocker run -it --gpus all -v /var/run/docker.sock:/var/run/docker.sock -v $(pwd):/workspace --net=host nvcr.io/nvidia/tritonserver:22.04-py3-sdk\nperf_analyzer -m model_torchscript -b 1 --concurrency-range 1:4\n```\n\n# model_analzyer\n\nTo use the `model-analyzer` shutdown previously started Triton server.\nEnter a docker container and run an analysis. All parameters for the\n`model-analyzer` are described here:\nhttps://github.com/triton-inference-server/model_analyzer/blob/main/docs/config.md.\n\n```bash\ndocker run -it --gpus all -v /var/run/docker.sock:/var/run/docker.sock -v $(pwd):$(pwd) --shm-size 1024m --net=host nvcr.io/nvidia/tritonserver:22.04-py3-sdk\ncd /home/... # go to the same path as You were in your host filesystem\nmodel-analyzer profile --model-repository $(pwd)/model_repository --profile-models model_onnx --triton-launch-mode=docker --output-model-repository-path $(pwd)/output/ -f perf.yaml --override-output-model-repository\n```\n\nTo generate a report follow instructions from `model-analyzer`.\n\n# Benchmark\n\nTo reproduce a benchmark reported in a blogpost run `run_benchmark.sh`\nscript in a `utils` directory. Use the same configuration as in the\n`perf_analyzer` section.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoftwaremill%2Ftriton_playground","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsoftwaremill%2Ftriton_playground","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoftwaremill%2Ftriton_playground/lists"}