{"id":15288143,"url":"https://github.com/seldonio/mlserver","last_synced_at":"2025-05-14T05:02:58.803Z","repository":{"id":36973551,"uuid":"272777821","full_name":"SeldonIO/MLServer","owner":"SeldonIO","description":"An inference server for your machine learning models, including support for multiple frameworks, multi-model serving and more","archived":false,"fork":false,"pushed_at":"2025-04-01T09:11:58.000Z","size":61527,"stargazers_count":795,"open_issues_count":158,"forks_count":193,"subscribers_count":25,"default_branch":"master","last_synced_at":"2025-04-03T11:57:31.963Z","etag":null,"topics":["kfserving","lightgbm","machine-learning","mlflow","scikit-learn","seldon-core","xgboost"],"latest_commit_sha":null,"homepage":"https://mlserver.readthedocs.io/en/latest/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SeldonIO.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-06-16T18:01:37.000Z","updated_at":"2025-04-02T08:07:17.000Z","dependencies_parsed_at":"2024-12-11T20:05:24.177Z","dependency_job_id":"d559e1d1-2166-48fd-ae8c-bd51a4effeec","html_url":"https://github.com/SeldonIO/MLServer","commit_stats":{"total_commits":1618,"total_committers":71,"mean_commits":"22.788732394366196","dds":0.5852904820766378,"last_synced_commit":"f5f1fee244ea8533d2b8368f3e4fea3b0a722a1f"},"previous_names":[],"tags_count":114,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SeldonIO%2FMLServer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SeldonIO%2FMLServer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SeldonIO%2FMLServer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SeldonIO%2FMLServer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SeldonIO","download_url":"https://codeload.github.com/SeldonIO/MLServer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248297821,"owners_count":21080312,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kfserving","lightgbm","machine-learning","mlflow","scikit-learn","seldon-core","xgboost"],"created_at":"2024-09-30T15:44:22.872Z","updated_at":"2025-04-10T20:54:28.403Z","avatar_url":"https://github.com/SeldonIO.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MLServer\n\nAn open source inference server for your machine learning models.\n\n[![video_play_icon](https://user-images.githubusercontent.com/10466106/151803854-75d17c32-541c-4eee-b589-d45b07ea486d.png)](https://www.youtube.com/watch?v=aZHe3z-8C_w)\n\n## Overview\n\nMLServer aims to provide an easy way to start serving your machine learning\nmodels through a REST and gRPC interface, fully compliant with [KFServing's V2\nDataplane](https://docs.seldon.io/projects/seldon-core/en/latest/reference/apis/v2-protocol.html)\nspec. Watch a quick video introducing the project [here](https://www.youtube.com/watch?v=aZHe3z-8C_w).\n\n- Multi-model serving, letting users run multiple models within the same\n  process.\n- Ability to run [inference in parallel for vertical\n  scaling](https://mlserver.readthedocs.io/en/latest/user-guide/parallel-inference.html)\n  across multiple models through a pool of inference workers.\n- Support for [adaptive\n  batching](https://mlserver.readthedocs.io/en/latest/user-guide/adaptive-batching.html),\n  to group inference requests together on the fly.\n- Scalability with deployment in Kubernetes native frameworks, including\n  [Seldon Core](https://docs.seldon.io/projects/seldon-core/en/latest/graph/protocols.html#v2-kfserving-protocol) and\n  [KServe (formerly known as KFServing)](https://kserve.github.io/website/modelserving/v1beta1/sklearn/v2/), where\n  MLServer is the core Python inference server used to serve machine learning\n  models.\n- Support for the standard [V2 Inference Protocol](https://docs.seldon.io/projects/seldon-core/en/latest/reference/apis/v2-protocol.html) on\n  both the gRPC and REST flavours, which has been standardised and adopted by\n  various model serving frameworks.\n\nYou can read more about the goals of this project on the [initial design\ndocument](https://docs.google.com/document/d/1C2uf4SaAtwLTlBCciOhvdiKQ2Eay4U72VxAD4bXe7iU/edit?usp=sharing).\n\n## Usage\n\nYou can install the `mlserver` package running:\n\n```bash\npip install mlserver\n```\n\nNote that to use any of the optional [inference runtimes](#inference-runtimes),\nyou'll need to install the relevant package.\nFor example, to serve a `scikit-learn` model, you would need to install the\n`mlserver-sklearn` package:\n\n```bash\npip install mlserver-sklearn\n```\n\nFor further information on how to use MLServer, you can check any of the\n[available examples](#examples).\n\n## Inference Runtimes\n\nInference runtimes allow you to define how your model should be used within\nMLServer.\nYou can think of them as the **backend glue** between MLServer and your machine\nlearning framework of choice.\nYou can read more about [inference runtimes in their documentation\npage](./docs/runtimes/index.md).\n\nOut of the box, MLServer comes with a set of pre-packaged runtimes which let\nyou interact with a subset of common frameworks.\nThis allows you to start serving models saved in these frameworks straight\naway.\nHowever, it's also possible to **[write custom\nruntimes](./docs/runtimes/custom.md)**.\n\nOut of the box, MLServer provides support for:\n\n| Framework     | Supported | Documentation                                                    |\n| ------------- | --------- | ---------------------------------------------------------------- |\n| Scikit-Learn  | ✅        | [MLServer SKLearn](./runtimes/sklearn)                           |\n| XGBoost       | ✅        | [MLServer XGBoost](./runtimes/xgboost)                           |\n| Spark MLlib   | ✅        | [MLServer MLlib](./runtimes/mllib)                               |\n| LightGBM      | ✅        | [MLServer LightGBM](./runtimes/lightgbm)                         |\n| CatBoost      | ✅        | [MLServer CatBoost](./runtimes/catboost)                         |\n| Tempo         | ✅        | [`github.com/SeldonIO/tempo`](https://github.com/SeldonIO/tempo) |\n| MLflow        | ✅        | [MLServer MLflow](./runtimes/mlflow)                             |\n| Alibi-Detect  | ✅        | [MLServer Alibi Detect](./runtimes/alibi-detect)                 |\n| Alibi-Explain | ✅        | [MLServer Alibi Explain](./runtimes/alibi-explain)               |\n| HuggingFace   | ✅        | [MLServer HuggingFace](./runtimes/huggingface)                   |\n\nMLServer is licensed under the Apache License, Version 2.0. However please note that software used in conjunction with, or alongside, MLServer may be licensed under different terms. For example, Alibi Detect and Alibi Explain are both licensed under the Business Source License 1.1. For more information about the legal terms of products that are used in conjunction with or alongside MLServer, please refer to their respective documentation.\n\n## Supported Python Versions\n\n🔴 Unsupported\n\n🟠 Deprecated: To be removed in a future version\n\n🟢 Supported\n\n🔵 Untested\n\n| Python Version | Status |\n| -------------- | ------ |\n| 3.7            | 🔴     |\n| 3.8            | 🔴     |\n| 3.9            | 🟢     |\n| 3.10           | 🟢     |\n| 3.11           | 🟢     |\n| 3.12           | 🟢     |\n| 3.13           | 🔴     |\n\n## Examples\n\nTo see MLServer in action, check out [our full list of\nexamples](./docs/examples/index.md).\nYou can find below a few selected examples showcasing how you can leverage\nMLServer to start serving your machine learning models.\n\n- [Serving a `scikit-learn` model](./docs/examples/sklearn/README.md)\n- [Serving a `xgboost` model](./docs/examples/xgboost/README.md)\n- [Serving a `lightgbm` model](./docs/examples/lightgbm/README.md)\n- [Serving a `catboost` model](./docs/examples/catboost/README.md)\n- [Serving a `tempo` pipeline](./docs/examples/tempo/README.md)\n- [Serving a custom model](./docs/examples/custom/README.md)\n- [Serving an `alibi-detect` model](./docs/examples/alibi-detect/README.md)\n- [Serving a `HuggingFace` model](./docs/examples/huggingface/README.md)\n- [Multi-Model Serving with multiple frameworks](./docs/examples/mms/README.md)\n- [Loading / unloading models from a model repository](./docs/examples/model-repository/README.md)\n\n## Developer Guide\n\n### Versioning\n\nBoth the main `mlserver` package and the [inference runtimes\npackages](./docs/runtimes/index.md) try to follow the same versioning schema.\nTo bump the version across all of them, you can use the\n[`./hack/update-version.sh`](./hack/update-version.sh) script.\n\nWe generally keep the version as a placeholder for an upcoming version.\n\nFor example:\n\n```bash\n./hack/update-version.sh 0.2.0.dev1\n```\n\n### Testing\n\nTo run all of the tests for MLServer and the runtimes, use:\n\n```bash\nmake test\n```\n\nTo run run tests for a single file, use something like:\n\n```bash\ntox -e py3 -- tests/batch_processing/test_rest.py\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fseldonio%2Fmlserver","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fseldonio%2Fmlserver","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fseldonio%2Fmlserver/lists"}