{"id":21757742,"url":"https://github.com/deepjavalibrary/djl-serving","last_synced_at":"2025-04-05T14:10:04.167Z","repository":{"id":37028849,"uuid":"396992684","full_name":"deepjavalibrary/djl-serving","owner":"deepjavalibrary","description":"A universal scalable machine learning model deployment solution","archived":false,"fork":false,"pushed_at":"2024-05-22T23:03:48.000Z","size":8805,"stargazers_count":173,"open_issues_count":56,"forks_count":56,"subscribers_count":12,"default_branch":"master","last_synced_at":"2024-05-22T23:10:05.481Z","etag":null,"topics":["deep-learning","deployment","djl","inference","pytorch","serving"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deepjavalibrary.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-08-16T22:16:36.000Z","updated_at":"2024-08-16T16:40:27.744Z","dependencies_parsed_at":"2023-12-18T17:05:37.789Z","dependency_job_id":"fd98d2b4-1484-4212-a3a5-cbfd47ede79b","html_url":"https://github.com/deepjavalibrary/djl-serving","commit_stats":{"total_commits":510,"total_committers":17,"mean_commits":30.0,"dds":0.5333333333333333,"last_synced_commit":"b048a278b87a236d4848190bd1b67f947910547c"},"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepjavalibrary%2Fdjl-serving","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepjavalibrary%2Fdjl-serving/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepjavalibrary%2Fdjl-serving/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepjavalibrary%2Fdjl-serving/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deepjavalibrary","download_url":"https://codeload.github.com/deepjavalibrary/djl-serving/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247345856,"owners_count":20924102,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","deployment","djl","inference","pytorch","serving"],"created_at":"2024-11-26T11:16:57.056Z","updated_at":"2025-04-05T14:10:04.133Z","avatar_url":"https://github.com/deepjavalibrary.png","language":"Java","funding_links":[],"categories":["人工智能"],"sub_categories":[],"readme":"# DJL Serving\n\n## Overview\n\nDJL Serving is a high performance universal stand-alone model serving solution powered by [DJL](https://djl.ai).\nIt takes a deep learning model, several models, or workflows and makes them available through an\nHTTP endpoint. It can serve the following model types out of the box:\n\n- PyTorch TorchScript model\n- TensorFlow SavedModel bundle\n- Apache MXNet model\n- ONNX model (CPU)\n- TensorRT model\n- Python script model\n\nYou can install extra extensions to enable the following models:\n\n- XGBoost model\n- LightGBM model\n- Sentencepiece model\n- fastText/BlazingText model\n\n## Key features\n\n- **Performance** - DJL serving running multithreading inference in a single JVM. Our benchmark shows\nDJL serving has higher throughput than most C++ model servers on the market\n- **Ease of use** - DJL serving can serve most models out of the box\n- **Easy to extend** - DJL serving plugins make it easy to add custom extensions\n- **Auto-scale** - DJL serving automatically scales up/down worker threads based on the load\n- **Dynamic batching** - DJL serving supports dynamic batching to increase throughput\n- **Model versioning** - DJL allows users to load different versions of a model on a single endpoint\n- **Multi-engine support** - DJL allows users to serve models from different engines at the same time\n\n## Installation\n\nFor macOS\n\n```\nbrew install djl-serving\n\n# Start djl-serving as service:\nbrew services start djl-serving\n\n# Stop djl-serving service\nbrew services stop djl-serving\n```\n\nFor Ubuntu\n\n```\ncurl -O https://publish.djl.ai/djl-serving/djl-serving_0.30.0-1_all.deb\nsudo dpkg -i djl-serving_0.30.0-1_all.deb\n```\n\nFor Windows\n\nWe are considering to create a `chocolatey` package for Windows. For the time being, you can\ndownload djl-serving zip file from [here](https://publish.djl.ai/djl-serving/serving-0.30.0.zip).\n\n```\ncurl -O https://publish.djl.ai/djl-serving/serving-0.30.0.zip\nunzip serving-0.30.0.zip\n# start djl-serving\nserving-0.30.0\\bin\\serving.bat\n```\n\n### Docker\n\nYou can also use docker to run DJL Serving:\n\n```\ndocker run -itd -p 8080:8080 deepjavalibrary/djl-serving\n```\n\n## Usage\n\nDJL Serving can be started from the command line.\nTo see examples, see the [starting page](serving/docs/starting.md).\n\n### More examples\n\n- [Serving a Python model](https://github.com/deepjavalibrary/djl-demo/tree/master/huggingface/python)\n- [Serving on Inferentia EC2 instance](https://github.com/deepjavalibrary/djl-demo/tree/master/huggingface/inferentia)\n- [Serving with docker](https://github.com/deepjavalibrary/djl-serving/tree/master/serving/docker)\n\n### More command line options\n\n```sh\ndjl-serving --help\nusage: djl-serving [OPTIONS]\n -f,--config-file \u003cCONFIG-FILE\u003e    Path to the configuration properties file.\n -h,--help                         Print this help.\n -m,--models \u003cMODELS\u003e              Models to be loaded at startup.\n -s,--model-store \u003cMODELS-STORE\u003e   Model store location where models can be loaded.\n -w,--workflows \u003cWORKFLOWS\u003e   Workflows to be loaded at startup.\n```\n\nSee [configuration](serving/docs/configuration.md) for more details about defining models, model-store, and workflows.\n\n## REST API\n\nDJL Serving uses a RESTful API for both inference and management calls.\n\nWhen DJL Serving starts up, it has two web services:\n\n* [Inference API](serving/docs/inference_api.md) - Used by clients to query the server and run models\n* [Management API](serving/docs/management_api.md) - Used to add, remove, and scale models on the server\n\nBy default, DJL Serving listens on port 8080 and is only accessible from localhost.\nPlease see [DJL Serving Configuration](serving/docs/configuration.md) for how to enable access from a remote host.\n\n## Architecture\n\nDetails about how DJL Serving is implemented can be found in the [architecture docs](serving/docs/architecture.md).\n\n# Plugin management\n\nDJL Serving supports plugins, user can implement their own plugins to enrich DJL Serving features.\nSee [DJL Plugin Management](serving/docs/plugin_management.md) for how to install plugins to DJL Serving.\n\n## Logging\nyou can set the logging level on the command-line adding a parameter for the JVM\n\n```sh\n-Dai.djl.logging.level={FATAL|ERROR|WARN|INFO|DEBUG|TRACE}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepjavalibrary%2Fdjl-serving","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeepjavalibrary%2Fdjl-serving","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepjavalibrary%2Fdjl-serving/lists"}