{"id":19719688,"url":"https://github.com/oeway/pyotritonclient","last_synced_at":"2025-04-13T17:33:11.125Z","repository":{"id":46004239,"uuid":"409242936","full_name":"oeway/pyotritonclient","owner":"oeway","description":"A Pyodide python http client library and utilities for communicating with Triton Inference Server (based on tritonclient from NVIDIA)","archived":false,"fork":false,"pushed_at":"2023-06-07T08:24:37.000Z","size":93,"stargazers_count":7,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-27T08:22:06.382Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oeway.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-09-22T14:45:58.000Z","updated_at":"2024-04-23T02:00:38.000Z","dependencies_parsed_at":"2023-02-17T01:55:15.460Z","dependency_job_id":null,"html_url":"https://github.com/oeway/pyotritonclient","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oeway%2Fpyotritonclient","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oeway%2Fpyotritonclient/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oeway%2Fpyotritonclient/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oeway%2Fpyotritonclient/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oeway","download_url":"https://codeload.github.com/oeway/pyotritonclient/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248752612,"owners_count":21156125,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T23:09:02.778Z","updated_at":"2025-04-13T17:33:11.075Z","avatar_url":"https://github.com/oeway.png","language":"Python","funding_links":[],"categories":["Easier Triton"],"sub_categories":[],"readme":"# Triton HTTP Client for Pyodide\n\nA Pyodide python http client library and utilities for communicating with Triton Inference Server (based on tritonclient from NVIDIA).\n\n\nThis is a simplified implemetation of the triton client from NVIDIA, it works both in the browser with Pyodide Python or the native Python.\nIt only implement the http client, and most of the API remains the similar but changed into async and with additional utility functions.\n\n## Installation\n\nTo use it in native CPython, you can install the package by running:\n```\npip install pyotritonclient\n```\n\nFor Pyodide-based Python environment, for example: [JupyterLite](https://jupyterlite.readthedocs.io/en/latest/_static/lab/index.html) or [Pyodide console](https://pyodide-cdn2.iodide.io/dev/full/console.html), you can install the client by running the following python code:\n```python\nimport micropip\nmicropip.install(\"pyotritonclient\")\n```\n## Usage\n\n### Basic example\nTo execute the model, we provide utility functions to make it much easier:\n```python\nimport numpy as np\nfrom pyotritonclient import execute\n\n# create fake input tensors\ninput0 = np.zeros([2, 349, 467], dtype='float32')\n# run inference\nresults = await execute(inputs=[input0, {\"diameter\": 30}], server_url='https://ai.imjoy.io/triton', model_name='cellpose-python')\n```\n\nThe above example assumes you are running the code in a jupyter notebook or an environment supports top-level await, if you are trying the example code in a normal python script, please wrap the code into an async function and run with asyncio as follows:\n```python\nimport asyncio\nimport numpy as np\nfrom pyotritonclient import execute\n\nasync def run():\n    results = await execute(inputs=[np.zeros([2, 349, 467], dtype='float32'), {\"diameter\": 30}], server_url='https://ai.imjoy.io/triton', model_name='cellpose-python')\n    print(results)\n\nloop = asyncio.get_event_loop()\nloop.run_until_complete(run())\n```\n\nYou can access the lower level api, see the [test example](./tests/test_client.py).\n\nYou can also find the official [client examples](https://github.com/triton-inference-server/client/tree/main/src/python/examples) demonstrate how to use the \npackage to issue request to [triton inference server](https://github.com/triton-inference-server/server). However, please notice that you will need to\nchange the http client code into async style. For example, instead of doing `client.infer(...)`, you need to do `await client.infer(...)`.\n\nThe http client code is forked from [triton client git repo](https://github.com/triton-inference-server/client) since commit [b3005f9db154247a4c792633e54f25f35ccadff0](https://github.com/triton-inference-server/client/tree/b3005f9db154247a4c792633e54f25f35ccadff0).\n\n\n### Using the sequence executor with stateful models\nTo simplify the manipulation on stateful models with sequence, we also provide the `SequenceExecutor` to make it easier to run models in a sequence.\n```python\nfrom pyotritonclient import SequenceExcutor\n\n\nseq = SequenceExcutor(\n  server_url='https://ai.imjoy.io/triton',\n  model_name='cellpose-train',\n  sequence_id=100\n)\ninputs = [\n  image.astype('float32'),\n  labels.astype('float32'),\n  {\"steps\": 1, \"resume\": True}\n]\nfor (image, labels, info) in train_samples:\n  result = await seq.step(inputs)\n\nresult = await seq.end(inputs)\n```\n\nNote that above example called `seq.end()` by sending the last inputs again to end the sequence. If you want to specify the inputs for the execution, you can run `result = await se.end(inputs)`.\n\nFor a small batch of data, you can also run it like this:\n```python\nfrom pyotritonclient import SequenceExcutor\n\nseq = SequenceExcutor(\n  server_url='https://ai.imjoy.io/triton',\n  model_name='cellpose-train',\n  sequence_id=100\n)\n\n# a list of inputs\ninputs_batch = [[\n  image.astype('float32'),\n  labels.astype('float32'),\n  {\"steps\": 1, \"resume\": True}\n] for (image, labels, _) in train_samples]\n\ndef on_step(i, result):\n  \"\"\"Function called on every step\"\"\"\n  print(i)\n\nresults = await seq(inputs_batch, on_step=on_step)\n```\n\n\n\n## Server setup\nSince we access the server from the browser environment which typically has more security restrictions, it is important that the server is configured to enable browser access.\n\nPlease make sure you configured following aspects:\n * The server must provide HTTPS endpoints instead of HTTP\n * The server should send the following headers:\n    - `Access-Control-Allow-Headers: Inference-Header-Content-Length,Accept-Encoding,Content-Encoding,Access-Control-Allow-Headers`\n    - `Access-Control-Expose-Headers: Inference-Header-Content-Length,Range,Origin,Content-Type`\n    - `Access-Control-Allow-Methods: GET,HEAD,OPTIONS,PUT,POST`\n    - `Access-Control-Allow-Origin: *` (This is optional depending on whether you want to support CORS)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foeway%2Fpyotritonclient","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foeway%2Fpyotritonclient","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foeway%2Fpyotritonclient/lists"}