{"id":20040779,"url":"https://github.com/vadimkantorov/tritonserverstringproc","last_synced_at":"2025-05-05T08:32:05.212Z","repository":{"id":196139743,"uuid":"694214284","full_name":"vadimkantorov/tritonserverstringproc","owner":"vadimkantorov","description":"Example string processing pipeline on Triton Inference Server","archived":false,"fork":false,"pushed_at":"2024-06-07T16:34:19.000Z","size":69,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-08T19:47:05.077Z","etag":null,"topics":["machine-learning-pipelines","python","triton-inference-server"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vadimkantorov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-20T14:50:02.000Z","updated_at":"2024-08-23T15:35:50.000Z","dependencies_parsed_at":"2024-06-07T17:11:36.705Z","dependency_job_id":null,"html_url":"https://github.com/vadimkantorov/tritonserverstringproc","commit_stats":null,"previous_names":["vadimkantorov/tritoninferernceprimer","vadimkantorov/tritonserverstringproc"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vadimkantorov%2Ftritonserverstringproc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vadimkantorov%2Ftritonserverstringproc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vadimkantorov%2Ftritonserverstringproc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vadimkantorov%2Ftritonserverstringproc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vadimkantorov","download_url":"https://codeload.github.com/vadimkantorov/tritonserverstringproc/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252466725,"owners_count":21752422,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["machine-learning-pipelines","python","triton-inference-server"],"created_at":"2024-11-13T10:43:48.878Z","updated_at":"2025-05-05T08:32:05.196Z","avatar_url":"https://github.com/vadimkantorov.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003e [!NOTE]\n\u003e Triton Inference Server's \"Generate\" extension might be a better choice for string processing endpoints:\n\u003e - https://github.com/triton-inference-server/server/issues/6337\n\u003e - https://github.com/triton-inference-server/server/pull/6412\n\u003e - https://github.com/triton-inference-server/server/issues/6745\n\u003e - https://github.com/triton-inference-server/server/blob/main/docs/protocol/extension_generate.md\n\u003e - https://blog.marvik.ai/2023/10/16/deploying-llama2-with-nvidia-triton-inference-server/\n\u003e - https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/protocol/extension_generate.html\n\u003e - https://github.com/triton-inference-server/server/blob/main/qa/python_models/string/model.py\n\n\n##  Primer of a string processing pipeline on Triton Inference Server on a CPU-only Docker-less system\n\nHere we have a few example Python models accepting a batch of JSON objects and returning a batch of JSON objects. These models are connected in a sequential **\"ensemble\"** pipeline (in Triton Inference Server lingo).\n\nIf you can't use the provided Docker images of `trintonserver`, please compile the `tritonserver` binary with Python backend [from source](https://github.com/triton-inference-server/server/blob/main/docs/customization_guide/build.md#cpu-only-build). See [buildtritoninferenceserver.yml](./.github/workflows/buildtritoninferenceserver.yml) for steps or use the GitHub Action / uploaded artifacts (for Ubuntu 22.04).\n\n```shell\n#e.g. download opt.zip from GitHub Actions build artifacts and link ./tritonserver/ to /opt/tritonserver/\n#sudo add-apt-repository -y ppa:mhier/libboost-latest \u0026\u0026 apt install libboost-filesystem1.81-dev # if not tritonserver not built statically with libboost_filesystem.a\n#unzip opt.zip\n#chmod +x \"$PWD/tritonserver/bin/tritonserver\"\n#sudo mkdir -p /opt/ \u0026\u0026 sudo ln -s \"$PWD/tritonserver\" /opt/tritonserver\n#export PATH=\"/opt/tritonserver/bin/:$PATH\"\n\ntritonserver --model-repository \"$PWD/models\" --log-verbose=1\n\ncurl -i http://localhost:8000/v2/health/ready\n# HTTP/1.1 200 OK\n\ncurl -i -X POST localhost:8000/v2/models/modelC/generate -d '{\"text_input\": \"Hello\"}'\n#HTTP/1.1 200 OK\n#Content-Type: application/json\n#Content-Type: application/json\n#Transfer-Encoding: chunked\n#{\"model_name\":\"modelC\",\"model_version\":\"1\",\"text_output\":\"modelC: Hello World\"}\n\ncurl -i -X POST localhost:8000/v2/models/modelC/infer --header 'Content-Type: application/json' --data-raw '{\"inputs\":[ { \"name\": \"text_input\", \"shape\": [1], \"datatype\": \"BYTES\", \"data\":  [\"Hello\"]  }  ] }'\n#HTTP/1.1 200 OK\n#Content-Type: application/json\n#Content-Length: 140\n#{\"model_name\":\"modelC\",\"model_version\":\"1\",\"outputs\":[{\"name\":\"text_output\",\"datatype\":\"BYTES\",\"shape\":[1],\"data\":[\"modelC: Hello World\"]}]}\n\ncurl -i -X POST localhost:8000/v2/models/modelA/infer -H 'Inference-Header-Content-Length: 140' -H \"Content-Type: application/octet-stream\" --data-binary '{\"inputs\":[{\"name\":\"INPUT0\",\"shape\":[15],\"datatype\":\"UINT8\",\"parameters\":{\"binary_data_size\":15}}],\"parameters\":{\"binary_data_output\":true}}{\"hi\": \"hello\"}'\n# HTTP/1.1 200 OK\n#Content-Type: application/octet-stream\n#Inference-Header-Content-Length: 143\n#Content-Length: 171\n#{\"model_name\":\"modelA\",\"model_version\":\"1\",\"outputs\":[{\"name\":\"OUTPUT0\",\"datatype\":\"UINT8\",\"shape\":[28],\"parameters\":{\"binary_data_size\":28}}]}{\"hi\": \"modelAmodelA:hello\"}\n\ncurl -i -X POST localhost:8000/v2/models/modelB/infer -H 'Inference-Header-Content-Length: 140' -H \"Content-Type: application/octet-stream\" --data-binary '{\"inputs\":[{\"name\":\"INPUT0\",\"shape\":[15],\"datatype\":\"UINT8\",\"parameters\":{\"binary_data_size\":15}}],\"parameters\":{\"binary_data_output\":true}}{\"hi\": \"hello\"}'\n# HTTP/1.1 200 OK\n#Content-Type: application/octet-stream\n#Inference-Header-Content-Length: 143\n#Content-Length: 171\n#{\"model_name\":\"modelB\",\"model_version\":\"1\",\"outputs\":[{\"name\":\"OUTPUT0\",\"datatype\":\"UINT8\",\"shape\":[28],\"parameters\":{\"binary_data_size\":28}}]}{\"hi\": \"modelBmodelB:hello\"}\n\ncurl -i -X POST localhost:8000/v2/models/pipeline/infer -H 'Inference-Header-Content-Length: 140' -H \"Content-Type: application/octet-stream\" --data-binary '{\"inputs\":[{\"name\":\"INPUT0\",\"shape\":[15],\"datatype\":\"UINT8\",\"parameters\":{\"binary_data_size\":15}}],\"parameters\":{\"binary_data_output\":true}}{\"hi\": \"hello\"}'\n#HTTP/1.1 200 OK\n#Content-Type: application/octet-stream\n#Inference-Header-Content-Length: 220\n#Content-Length: 261\n#{\"model_name\":\"pipeline\",\"model_version\":\"1\",\"parameters\":{\"sequence_id\":0,\"sequence_start\":false,\"sequence_end\":false},\"outputs\":[{\"name\":\"OUTPUT0\",\"datatype\":\"UINT8\",\"shape\":[41],\"parameters\":{\"binary_data_size\":41}}]}{\"hi\": \"modelBmodelB:modelAmodelA:hello\"}\n\n# the requests above can also be generated by\npython3 client.py modelA hello1\npython3 client.py modelB hello2\npython3 client.py pipeline hello3\n```\n\n## References\n- https://github.com/triton-inference-server/server/blob/main/docs/protocol/extension_generate.md\n- https://github.com/triton-inference-server/server/pull/6412\n- https://github.com/triton-inference-server/python_backend\n- https://github.com/triton-inference-server/python_backend/tree/main/examples/preprocessing\n- https://github.com/triton-inference-server/python_backend/tree/main/examples/auto_complete\n- https://blog.ml6.eu/triton-ensemble-model-for-deploying-transformers-into-production-c0f727c012e3\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvadimkantorov%2Ftritonserverstringproc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvadimkantorov%2Ftritonserverstringproc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvadimkantorov%2Ftritonserverstringproc/lists"}