{"id":26154503,"url":"https://github.com/datacanvasio/lms","last_synced_at":"2025-10-08T10:49:36.546Z","repository":{"id":204873185,"uuid":"711873255","full_name":"DataCanvasIO/LMS","owner":"DataCanvasIO","description":null,"archived":false,"fork":false,"pushed_at":"2023-11-06T12:22:11.000Z","size":9465,"stargazers_count":17,"open_issues_count":0,"forks_count":3,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-09-25T00:33:51.538Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DataCanvasIO.png","metadata":{"files":{"readme":"README.MD","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-30T10:50:32.000Z","updated_at":"2024-01-03T10:09:36.000Z","dependencies_parsed_at":"2025-03-11T08:41:33.851Z","dependency_job_id":null,"html_url":"https://github.com/DataCanvasIO/LMS","commit_stats":null,"previous_names":["datacanvasio/lms"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/DataCanvasIO/LMS","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataCanvasIO%2FLMS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataCanvasIO%2FLMS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataCanvasIO%2FLMS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataCanvasIO%2FLMS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DataCanvasIO","download_url":"https://codeload.github.com/DataCanvasIO/LMS/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataCanvasIO%2FLMS/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278931654,"owners_count":26070788,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-11T08:30:00.874Z","updated_at":"2025-10-08T10:49:36.526Z","avatar_url":"https://github.com/DataCanvasIO.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n    \u003ch1 align=\"center\"\u003eLMS - Large Model Serving\u003c/h1\u003e\n\u003c/div\u003e\n\n## Introduction\n\nLMS（Large Model Serving） is an open source tool that provides large model services. LMS can provide model compression, model evaluation,\nmodel deployment, model monitoring and other functions. \n\nIt includes LMS_Web and LMS_Client. In LMS_Client, you can\nimport, compress and deploy models through the command line. LMS_Web provides basic model information and a visual\ninterface for monitoring deployed models.\n\n![LMS Architecture](assets/image.png)\n\n## Installation and setting\n\n### Installation\n\nYou can install LMS with the following command.\n\n```bash\npip install dc-lms\n```\n\nor\n\n```bash\ngit clone git@github.com:DataCanvasIO/LMS.git\ncd lms\npython setup.py install\n```\n\n### Start \u0026 Stop lms_web\n\nYou can start LMS_Web with the following command.\n\n```bash\nlms_web start --port {{LMS_WEB_PORT}}\n```\n\nYou can stop LMS_Web with the following command.\n\n```bash\nlms_web stop\n```\n\n### Setting\n\nYou should initiate client using following command which will register self into the web center and start a daemon to\nexpose the monitoring metrics.\n\n```bash\nlms join {{LMS_WEB_HOST_NAME}}:{{LMS_WEB_PORT}}\n```\n\n## Quick Guide\n\n### Model List\n\n1. Add model\n   You can add the model to LMS through the following command.\n\n    ```bash\n    lms import --model_path {{model_path}}\n    ```\n\n   After adding the model successfully,You can view the model in LMS_Web.\n\n2. Delete model\n   You can delete the model from LMS through the following command.\n\n    ```bash\n    lms del --model_name {{model_name}}\n    ```\n\n3. List model\n   You can list the model of the current node from the LMS with the following command.\n\n    ```bash\n    lms list\n    ```\n\n### Model Evaluation\n\n1. Automatic evaluation:\n   the preset task will be used to test the specified model. The preset task includes `MMLU`, `CMMLU`, `BigBench`, `ARC`\n   , `AGIEval`, `ceval` and other benchmark. The model will be evaluated from multiple angles. You can automatically\n   evaluate the model with the following instructions.\n    ```bash\n    lms eval --model_name {{model_name}} --task CMMLU,ceval,ARC --output_path {{output_path}}\n    ```\n\n2. Custom evaluation\n   The system will use your specified task to test the specified model, and you can make a custom evaluation with the\n   following command.\n\n    ```bash\n    lms eval --model_name {{model_name}} --task {{custom_task.py}} --input_path {{input_path}} --output_path {{output_path}}\n    ```\n\n3. Manual evaluation\n   Execute the following command, and the system will test the model with the data you specify and return the model\n   results to the specified path.\n\n    ```bash\n    lms eval --model_name {{model_name}} --task human --input_path {{input_path}} --output_path {{output_path}}\n    ```\n\n   After the execution is successful, the model output will be displayed on the LMS_Web corresponding model details\n   page. You need to manually evaluate the output of the model.\n\nAfter the evaluation is successful, the evaluation result will be displayed on the LMS_Web corresponding model details\npage.\n\n### Quantization\n\nYou can quantize the specified model with the following command.\n\n```bash\nlms quantization --model_name {{model_name}} --{{int8|int4}} --quantized_model_path {{quantized_model_path}}\n```\n\n\u003e See examples for more details.\n\nAfter the quantization of the model is completed, a new quantized model is generated, and you can view the new model in\nLMS_Web.\n\n### Pruning\n\nYou can prune the specified model with the following command.\n\n```bash\nlms pruning {{sparse|structure}} --model_name {{model_name}}  --pruned_model_path {{pruned_model_path}}\n```\n\nAfter the pruning of the model is completed, a new pruned model is generated, and you can view the new model in LMS_Web.\n\n### Model Deployment\n\nYou can deploy bloom, llama, falcon, and many other models using the following commands.\n\n1. deploy supported model\n    ```bash\n    lms deploy --model_name {{model_name}} --gpu 0,1 --load_{{fp16|int8|int4}} --infer_config infer_conf.json\n    ```\n\n   If you need to deploy unsupported model, you can use deploy custom model.\n\n2. deploy custom model\n    ```bash\n    lms deploy --model_name {{model_name}} --gpu 0,1 --load_{{fp16|int8|int4}} --infer_py generate.py --infer_config infer_conf.json\n    ```\n\n\u003e See examples for more details on infer_config and infer_py.\n\nIf you want to deploy with specific port, use flag:`--port {{port}}` at above commands.\n\nAfter the model is deployed successfully, you can view the deployment status and monitoring of the model in LMS_Web.\n\n1. undeploy model\n\n   You can undeploy the mode with the following command.\n\n    ```bash\n    lms undeploy --model_name {{model_name}}\n    ```\n\n2. deployment logs\n\n   If you want to watch the log of the deployed model, you can use the following command.\n   ```bash\n   lms logs -f --model_name {{model_name}}\n   ```\n\n## Supported Models\n\n| ModelName | Quantize int8 | Quantize int4 | Prune sparse | Prune structure | Deploy fp16 | Deploy int8 | Deploy int4 |\n|-----------|---------------|---------------|--------------|-----------------|-------------|-------------|-------------|\n| Alaya    | Y             | Y             | Y            | Y               | Y           | Y           | Y           |\n| llama2    | Y             | Y             | Y            | Y               | Y           | Y           | Y           |\n| RedPajama | Y             | Y             | Y            | ❌               | Y           | Y           | Y           |\n| chatyuan  | Y             | ❌             | ❌            | ❌               | Y           | Y           | Y           |\n| bloom     | Y             | Y             | Y            | Y               | Y           | Y           | Y           |\n| falcon    | Y             | Y             | Y            | ❌               | Y           | Y           | Y           |\n| mpt       | Y             | Y             | Y            | ❌               | Y           | Y           | Y           |\n| GPT-J     | Y             | ❌             | Y            | ❌               | Y           | Y           | Y           |\n| dolly     | Y             | Y             | Y            | ❌               | Y           | Y           | Y           |\n| T5        | Y             | ❌             | ❌            | ❌               | Y           | Y           | Y           |\n\n## License\n\nThis project is released under the [Apache 2.0 license](LICENSE).\n\n## DataCanvas\nLMS is an open source project created by [DataCanvas](https://www.datacanvas.com/). \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatacanvasio%2Flms","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatacanvasio%2Flms","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatacanvasio%2Flms/lists"}