{"id":13398604,"url":"https://github.com/tensorflow/serving","last_synced_at":"2025-05-12T11:20:56.752Z","repository":{"id":37444175,"uuid":"50461701","full_name":"tensorflow/serving","owner":"tensorflow","description":"A flexible, high-performance serving system for machine learning models","archived":false,"fork":false,"pushed_at":"2025-05-11T07:02:23.000Z","size":19760,"stargazers_count":6277,"open_issues_count":115,"forks_count":2200,"subscribers_count":229,"default_branch":"master","last_synced_at":"2025-05-11T11:12:09.188Z","etag":null,"topics":["cpp","deep-learning","deep-neural-networks","machine-learning","ml","neural-network","python","serving","tensorflow"],"latest_commit_sha":null,"homepage":"https://www.tensorflow.org/serving","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tensorflow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2016-01-26T21:48:20.000Z","updated_at":"2025-05-11T07:02:28.000Z","dependencies_parsed_at":"2024-01-22T16:02:28.350Z","dependency_job_id":"e52439ce-6830-43c3-a3b3-012ef45f3922","html_url":"https://github.com/tensorflow/serving","commit_stats":{"total_commits":8364,"total_committers":230,"mean_commits":36.36521739130435,"dds":"0.17216642754662836","last_synced_commit":"648c9ee6489a3cf820aa1fcab82b821209e82af5"},"previous_names":[],"tags_count":126,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tensorflow%2Fserving","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tensorflow%2Fserving/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tensorflow%2Fserving/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tensorflow%2Fserving/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tensorflow","download_url":"https://codeload.github.com/tensorflow/serving/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253554098,"owners_count":21926614,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cpp","deep-learning","deep-neural-networks","machine-learning","ml","neural-network","python","serving","tensorflow"],"created_at":"2024-07-30T19:00:29.184Z","updated_at":"2025-05-11T11:12:19.627Z","avatar_url":"https://github.com/tensorflow.png","language":"C++","readme":"# TensorFlow Serving\n\n[![Ubuntu Build Status](https://storage.googleapis.com/tensorflow-serving-kokoro-build-badges-bucket/ubuntu.svg)](https://storage.googleapis.com/tensorflow-serving-kokoro-build-badges-bucket/ubuntu.html)\n[![Ubuntu Build Status at TF HEAD](https://storage.googleapis.com/tensorflow-serving-kokoro-build-badges-bucket/ubuntu-tf-head.svg)](https://storage.googleapis.com/tensorflow-serving-kokoro-build-badges-bucket/ubuntu-tf-head.html)\n![Docker CPU Nightly Build Status](https://storage.googleapis.com/tensorflow-serving-kokoro-build-badges-bucket/docker-cpu-nightly.svg)\n![Docker GPU Nightly Build Status](https://storage.googleapis.com/tensorflow-serving-kokoro-build-badges-bucket/docker-gpu-nightly.svg)\n\n----\nTensorFlow Serving is a flexible, high-performance serving system for\nmachine learning models, designed for production environments. It deals with\nthe *inference* aspect of machine learning, taking models after *training* and\nmanaging their lifetimes, providing clients with versioned access via\na high-performance, reference-counted lookup table.\nTensorFlow Serving provides out-of-the-box integration with TensorFlow models,\nbut can be easily extended to serve other types of models and data.\n\nTo note a few features:\n\n-   Can serve multiple models, or multiple versions of the same model\n    simultaneously\n-   Exposes both gRPC as well as HTTP inference endpoints\n-   Allows deployment of new model versions without changing any client code\n-   Supports canarying new versions and A/B testing experimental models\n-   Adds minimal latency to inference time due to efficient, low-overhead\n    implementation\n-   Features a scheduler that groups individual inference requests into batches\n    for joint execution on GPU, with configurable latency controls\n-   Supports many *servables*: Tensorflow models, embeddings, vocabularies,\n    feature transformations and even non-Tensorflow-based machine learning\n    models\n\n## Serve a Tensorflow model in 60 seconds\n```bash\n# Download the TensorFlow Serving Docker image and repo\ndocker pull tensorflow/serving\n\ngit clone https://github.com/tensorflow/serving\n# Location of demo models\nTESTDATA=\"$(pwd)/serving/tensorflow_serving/servables/tensorflow/testdata\"\n\n# Start TensorFlow Serving container and open the REST API port\ndocker run -t --rm -p 8501:8501 \\\n    -v \"$TESTDATA/saved_model_half_plus_two_cpu:/models/half_plus_two\" \\\n    -e MODEL_NAME=half_plus_two \\\n    tensorflow/serving \u0026\n\n# Query the model using the predict API\ncurl -d '{\"instances\": [1.0, 2.0, 5.0]}' \\\n    -X POST http://localhost:8501/v1/models/half_plus_two:predict\n\n# Returns =\u003e { \"predictions\": [2.5, 3.0, 4.5] }\n```\n\n## End-to-End Training \u0026 Serving Tutorial\n\nRefer to the official Tensorflow documentations site for [a complete tutorial to train and serve a Tensorflow Model](https://www.tensorflow.org/tfx/tutorials/serving/rest_simple).\n\n\n## Documentation\n\n### Set up\n\nThe easiest and most straight-forward way of using TensorFlow Serving is with\nDocker images. We highly recommend this route unless you have specific needs\nthat are not addressed by running in a container.\n\n*   [Install Tensorflow Serving using Docker](tensorflow_serving/g3doc/docker.md)\n    *(Recommended)*\n*   [Install Tensorflow Serving without Docker](tensorflow_serving/g3doc/setup.md)\n    *(Not Recommended)*\n*   [Build Tensorflow Serving from Source with Docker](tensorflow_serving/g3doc/building_with_docker.md)\n*   [Deploy Tensorflow Serving on Kubernetes](tensorflow_serving/g3doc/serving_kubernetes.md)\n\n### Use\n\n#### Export your Tensorflow model\n\nIn order to serve a Tensorflow model, simply export a SavedModel from your\nTensorflow program.\n[SavedModel](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/saved_model/README.md)\nis a language-neutral, recoverable, hermetic serialization format that enables\nhigher-level systems and tools to produce, consume, and transform TensorFlow\nmodels.\n\nPlease refer to [Tensorflow documentation](https://www.tensorflow.org/guide/saved_model#save_and_restore_models)\nfor detailed instructions on how to export SavedModels.\n\n#### Configure and Use Tensorflow Serving\n\n* [Follow a tutorial on Serving Tensorflow models](tensorflow_serving/g3doc/serving_basic.md)\n* [Configure Tensorflow Serving to make it fit your serving use case](tensorflow_serving/g3doc/serving_config.md)\n* Read the [Performance Guide](tensorflow_serving/g3doc/performance.md)\nand learn how to [use TensorBoard to profile and optimize inference requests](tensorflow_serving/g3doc/tensorboard.md)\n* Read the [REST API Guide](tensorflow_serving/g3doc/api_rest.md)\nor [gRPC API definition](https://github.com/tensorflow/serving/tree/master/tensorflow_serving/apis)\n* [Use SavedModel Warmup if initial inference requests are slow due to lazy initialization of graph](tensorflow_serving/g3doc/saved_model_warmup.md)\n* [If encountering issues regarding model signatures, please read the SignatureDef documentation](tensorflow_serving/g3doc/signature_defs.md)\n* If using a model with custom ops, [learn how to serve models with custom ops](tensorflow_serving/g3doc/custom_op.md)\n\n### Extend\n\nTensorflow Serving's architecture is highly modular. You can use some parts\nindividually (e.g. batch scheduling) and/or extend it to serve new use cases.\n\n* [Ensure you are familiar with building Tensorflow Serving](tensorflow_serving/g3doc/building_with_docker.md)\n* [Learn about Tensorflow Serving's architecture](tensorflow_serving/g3doc/architecture.md)\n* [Explore the Tensorflow Serving C++ API reference](https://www.tensorflow.org/tfx/serving/api_docs/cc/)\n* [Create a new type of Servable](tensorflow_serving/g3doc/custom_servable.md)\n* [Create a custom Source of Servable versions](tensorflow_serving/g3doc/custom_source.md)\n\n## Contribute\n\n\n**If you'd like to contribute to TensorFlow Serving, be sure to review the\n[contribution guidelines](CONTRIBUTING.md).**\n\n\n## For more information\n\nPlease refer to the official [TensorFlow website](http://tensorflow.org) for\nmore information.\n","funding_links":[],"categories":["C++","🎯 Tool Categories","\u003ca name=\"cpp\"\u003e\u003c/a\u003eC++","Serving","Frameworks/Servers for Serving","Deep Learning Framework","Deployment and Serving","其他_机器学习与深度学习","Application Recommendation","Software","Projects built with Bazel","Inference \u0026 Serving"],"sub_categories":["🏆 Top Serving Platforms","Frameworks/Servers for Serving","High-Level DL APIs","🧠 AI Applications","Deploying models","Google projects","Model Serving Frameworks"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftensorflow%2Fserving","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftensorflow%2Fserving","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftensorflow%2Fserving/lists"}