{"id":34086780,"url":"https://github.com/mam-dev/pydantic-open-inference","last_synced_at":"2026-03-11T13:40:20.948Z","repository":{"id":324223808,"uuid":"1094882464","full_name":"mam-dev/pydantic-open-inference","owner":"mam-dev","description":"Call an Open Inference server using pydantic models defining your model inputs and outputs.","archived":false,"fork":false,"pushed_at":"2025-11-19T12:37:04.000Z","size":165,"stargazers_count":1,"open_issues_count":3,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-16T16:24:42.255Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mam-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-12T09:52:10.000Z","updated_at":"2025-11-19T12:37:07.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/mam-dev/pydantic-open-inference","commit_stats":null,"previous_names":["mam-dev/pydantic-open-inference"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mam-dev/pydantic-open-inference","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mam-dev%2Fpydantic-open-inference","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mam-dev%2Fpydantic-open-inference/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mam-dev%2Fpydantic-open-inference/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mam-dev%2Fpydantic-open-inference/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mam-dev","download_url":"https://codeload.github.com/mam-dev/pydantic-open-inference/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mam-dev%2Fpydantic-open-inference/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30382674,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-11T12:49:11.341Z","status":"ssl_error","status_checked_at":"2026-03-11T12:46:41.342Z","response_time":84,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-12-14T13:33:08.569Z","updated_at":"2026-03-11T13:40:20.943Z","avatar_url":"https://github.com/mam-dev.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pydantic-open-inference\n\n[![image](https://img.shields.io/pypi/v/pydantic-open-inference\n)](https://pypi.python.org/pypi/pydantic-open-inference)\n[![image](https://img.shields.io/github/license/mam-dev/pydantic-open-inference\n)](https://github.com/mam-dev/pydantic-open-inference/blob/main/LICENSE)\n[![image](https://img.shields.io/pypi/pyversions/pydantic-open-inference)](https://pypi.python.org/pypi/pydantic-open-inference)\n[![Actions status](https://github.com/mam-dev/pydantic-open-inference/actions/workflows/ci.yaml/badge.svg)](https://github.com/mam-dev/pydantic-open-inference/actions)\n[![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://github.com/astral-sh/uv)\n[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)\n[![Mypy](https://www.mypy-lang.org/static/mypy_badge.svg)](https://www.mypy-lang.org)\n\n## What is it?\nProvides a client-side python interface for calling [open inference servers](https://kserve.github.io/website/docs/concepts/architecture/data-plane/v2-protocol).\nDefine [pydantic](https://docs.pydantic.dev/latest/) models for the model inputs and outputs, and it gets automatically\nconverted to and from the open inference protocol, removing the need for boilerplate\ncoding handling raw tensors and their shapes.\n\n## Features\n* ✅ Simple, type-safe API toward an open inference server using user-defined pydantic models\n* ✅ Automatic shape inference from pydantic models\n* ✅ Automatic datatype mapping (`int`→INT64, `float`→FP32, `str`→BYTES)\n* ✅ Datatype override for fine-grained control of model inputs\n* ✅ Automatically request only a subset of model outputs by omitting fields in your pydantic model\n* ✅ Take advantage of the full power of pydantic to validate and type-coerce your model calls\n* ✅ Multi-input/output models - each input/output maps to a pydantic field\n* ✅ Custom timeout configuration - configurable per model\n* ✅ Support calling inference for versioned and unversioned models\n* ✅ Error handling with custom exceptions - handle, e.g., specific error codes from the inference server differently\n* ✅ Fully typed - use, e.g., [mypy](https://mypy-lang.org/) and its [pydantic plugin](https://docs.pydantic.dev/latest/integrations/mypy/) to type check your code\n* ✅ Thread safe\n\n## Installation\nInstallation using [pip](https://github.com/pypa/pip):\n```bash\npip install pydantic-open-inference\n```\n\nAdding it to your project dependencies using [uv](https://docs.astral.sh/uv/):\n```bash\nuv add pydantic-open-inference\n```\n\n## How to use it\nAssume we have an inference server serving an Open Inference V2 REST API at `http://localhost:8080`\nand that the server contains a model named `example_model`, which has the following model **inputs**\n\n| name | shape | datatype | description              | example data       |\n|------|-------|----------|--------------------------|--------------------|\n| text | [1]   | BYTES    | A single string of text. | `[\"This is a text\"]` |\n\nand the following model **outputs**:\n\n| name     | shape   | datatype | description                                                                                                                                                                                                                                  | example data                                                     |\n|----------|---------|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------|\n| entities | [-1, 4] | BYTES    | \u003cdiv style=\"width:200px\"\u003eA list of entities extracted from the text. Each entity is made up of (in order) a label (either \"order-id\" or \"invoice-id\"), a confidence score, a start index for the entity in the text, and an end index.\u003c/div\u003e | `[[\"order-id\", \"0.9\", \"23\", \"30\"],[\"invoice-id\",\"0.8\",\"34\",\"46\"]]` |\n\nWe then define a pydantic model for the inputs, inheriting from `InputsBaseModel`.\nEach model input name needs to correspond to a field in the pydantic model, and the type of the\nfield needs to make sense given the shape and datatype of the input. In this case, there\nis just a single input named \"text\" that is supposed to hold a single text, so we simply define\n```python\nfrom pydantic_open_inference import InputsBaseModel\n\nclass ExampleModelInputs(InputsBaseModel):\n    text: str\n\n```\n\nIn the same way, we define a pydantic model for the outputs, inheriting from `OutputsBaseModel`.\nThere is one model outputs named \"entities\", so that will be the only field we define.\nThere can be any number of entities (corresponding to the -1 in the shape), so we define it to\nbe a `list`. A list of what? Here we have some choices. We could simply define `entities: list[list[str]]`\nand get each entity as a list of strings, e.g., `[\"order-id\", \"0.98\", \"12\", \"24\"]`, but this is not\nvery useful to us, so we instead define, e.g., a _named tuple_ with ordered, named, and typed fields:\n```python\nfrom typing import NamedTuple\nfrom enum import StrEnum\nfrom pydantic_open_inference import OutputsBaseModel\n\nclass Label(StrEnum):\n    ORDER_ID = \"order-id\"\n    INVOICE_ID = \"invoice-id\"\n\nclass Entity(NamedTuple):\n    label: Label\n    score: float\n    start: int\n    end: int\n\nclass ExampleModelOutputs(OutputsBaseModel):\n    entities: list[Entity]\n```\nNote that we could also have used `typing.Literal` instead of `enum.StrEnum`, and even added further validation,\ne.g., that the score is between 0.0 and 1.0, and that the start index is not negative etc. Pydantic offers many\npossibilities, but for now, let us stick with this.\n\nWe now create a `RemoteModel` instance. The `RemoteModel` class represents a model in the inference server,\nin this case our \"example_model\". Instantiation requires the URL of the inference server, the name of the\nmodel, and the pydantic models we defined for the input and output:\n```python\nfrom pydantic_open_inference import RemoteModel\n\nINFERENCE_SERVER_URL = \"http://localhost:8080\"\n\nexample_model = RemoteModel(\n    server_url=INFERENCE_SERVER_URL,\n    model_name=\"example_model\",\n    inputs_model=ExampleModelInputs,\n    outputs_model=ExampleModelOutputs,\n)\n```\n\nIn order to call the inference server, we simply call the `RemoteModel.infer` method, giving\nit an instance of `ExampleModelInputs`, and get back an instance of `ExampleModelOutputs`:\n```python\nexample_input = ExampleModelInputs(text=\"Your order id is 123456.\")\nexample_output = example_model.infer(example_input)\n# ExampleModelOutputs(\n#    entities=[\n#        Entity(label=Label.ORDER_ID, score=0.95, start=17, end=23),\n#    ]\n# )\n```\nWe have thus not only made a perfectly type-safe and validated call to the inference server,\nwe have also gotten the response back in a useful format! We never had to bother with\nhandling the shape of the data, parsing it in row-major order etc, as all of that is handled under the hood!\n\nThe only time you need to think about \"shape\" is when defining the fields of the pydantic models.\nUse types like, e.g., `list`, `tuple`, and `typing.NamedTuple`, to type the pydantic-model fields\nso that they map sensibly to their corresponding model input/output. Each input/output name must\nmatch to a pydantic field and the shape must correspond to the structure of the field, e.g.,\n`list[tuple[int, int]]` for shape [-1, 2], or `tuple[tuple[float, float], tuple[float, float]]`\nfor shape [2, 2], etc.\n\nThe \"datatype\" of the model _inputs_ is automatically set based on the innermost type of the\ncorresponding field in the `InputsBaseModel` subclass, according to:\n\n| python type | datatype |\n|-------------|----------|\n| bool        | BOOL     |\n| int         | INT64    |\n| float       | FP32     |\n| str         | BYTES    |\n\nHowever, you can always _override_ the datatype of the model inputs to whatever you want using\n`pydantic_open_inference.DatatypeOverride`; see its docstring for more information.\nThe datatypes of the model _outputs_ are disregarded; all that matters is \nthat the data we got back from the model can be mapped to the defined `OutputsBaseModel` subclass.\n\nWhen an inference call to a model is made, it automatically requests only the specific outputs\nfor which you have defined fields in your `OutputsBaseModel` subclass. So if you are not\ninterested in an output, simply do not define a pydantic field for it.\n\n## Examples\n\nSee the [examples directory](examples) for more examples.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmam-dev%2Fpydantic-open-inference","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmam-dev%2Fpydantic-open-inference","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmam-dev%2Fpydantic-open-inference/lists"}