{"id":13637763,"url":"https://github.com/microsoft/lmops","last_synced_at":"2025-05-13T21:04:43.040Z","repository":{"id":64760141,"uuid":"577753426","full_name":"microsoft/LMOps","owner":"microsoft","description":"General technology for enabling AI capabilities w/ LLMs and MLLMs","archived":false,"fork":false,"pushed_at":"2025-04-16T16:24:39.000Z","size":75484,"stargazers_count":3973,"open_issues_count":67,"forks_count":308,"subscribers_count":56,"default_branch":"main","last_synced_at":"2025-05-07T23:47:01.389Z","etag":null,"topics":["agi","gpt","language-model","llm","lm","lmops","nlp","pretraining","prompt","promptist","x-prompt"],"latest_commit_sha":null,"homepage":"https://aka.ms/GeneralAI","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/microsoft.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":"SUPPORT.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-12-13T12:56:21.000Z","updated_at":"2025-05-07T13:02:21.000Z","dependencies_parsed_at":"2025-04-28T12:23:39.112Z","dependency_job_id":null,"html_url":"https://github.com/microsoft/LMOps","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FLMOps","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FLMOps/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FLMOps/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FLMOps/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/microsoft","download_url":"https://codeload.github.com/microsoft/LMOps/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254028482,"owners_count":22002268,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agi","gpt","language-model","llm","lm","lmops","nlp","pretraining","prompt","promptist","x-prompt"],"created_at":"2024-08-02T01:00:30.317Z","updated_at":"2025-05-13T21:04:38.009Z","avatar_url":"https://github.com/microsoft.png","language":"Python","funding_links":[],"categories":["Models","Prompts"],"sub_categories":["Projects"],"readme":"\u003c!--# LMOps: Enabling AI w/ LLMs--\u003e\n\n# LMOps\nLMOps is a research initiative on fundamental research and technology for building AI products w/ foundation models, especially on the general technology for enabling AI capabilities w/ LLMs and Generative AI models.\n\n- Better Prompts: [Automatic Prompt Optimization](https://arxiv.org/abs/2305.03495), [Promptist](https://arxiv.org/abs/2212.09611), [Extensible prompts](https://arxiv.org/abs/2212.00616), [Universal prompt retrieval](https://arxiv.org/abs/2303.08518), [LLM Retriever](https://arxiv.org/abs/2307.07164), [In-Context Demonstration Selection](https://arxiv.org/abs/2305.14726)\n- Longer Context: [Structured prompting](https://arxiv.org/abs/2212.06713), [Length-Extrapolatable Transformers](https://arxiv.org/abs/2212.10554)\n- LLM Alignment: [Alignment via LLM feedback]()\n- LLM Accelerator (Faster Inference): [Lossless Acceleration of LLMs](https://arxiv.org/abs/2304.04487)\n- LLM Customization: [Adapt LLM to domains](https://arxiv.org/pdf/2309.09530.pdf)\n- Fundamentals: [Understanding In-Context Learning](https://arxiv.org/abs/2212.10559)\n\n## Links\n\n- [microsoft/unilm](https://github.com/microsoft/unilm): Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities\n- [microsoft/torchscale](https://github.com/microsoft/torchscale): Transformers at (any) Scale\n\n## News\n- [Paper Release] Nov, 2023: [In-Context Demonstration Selection with Cross Entropy Difference](https://arxiv.org/abs/2305.14726) (EMNLP 2023)\n- [Paper Release] Oct, 2023: [Tuna: Instruction Tuning using Feedback from Large Language Models](https://arxiv.org/pdf/2310.13385.pdf) (EMNLP 2023)\n- [Paper Release] Oct, 2023: [Automatic Prompt Optimization with \"Gradient Descent\" and Beam Search](https://arxiv.org/abs/2305.03495) (EMNLP 2023)\n- [Paper Release] Oct, 2023: [UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation](https://arxiv.org/abs/2303.08518) (EMNLP 2023)\n- [Paper Release] July, 2023: [Learning to Retrieve In-Context Examples for Large Language Models](https://arxiv.org/abs/2307.07164)\n- [Paper Release] April, 2023: [Inference with Reference: Lossless Acceleration of Large Language Models](https://arxiv.org/abs/2304.04487)\n- [Paper Release] Dec, 2022: [Why Can GPT Learn In-Context? Language Models Secretly Perform Finetuning as Meta Optimizers](https://arxiv.org/abs/2212.10559)\n- [Paper \u0026 Model \u0026 Demo Release] Dec, 2022: [Optimizing Prompts for Text-to-Image Generation](https://aka.ms/promptist)\n- [Paper \u0026 Code Release] Dec, 2022: [Structured Prompting: Scaling In-Context Learning to 1,000 Examples](https://arxiv.org/abs/2212.06713)\n- [Paper Release] Nov, 2022: [Extensible Prompts for Language Models](https://arxiv.org/abs/2212.00616)\n\n## Prompt Intelligence\n\nAdvanced technologies facilitating prompting language models.\n\n### Promptist: reinforcement learning for automatic prompt optimization\n\n[Paper] [Optimizing Prompts for Text-to-Image Generation](https://arxiv.org/abs/2212.09611)\n\n\u003e - Language models serve as a prompt interface that optimizes user input into model-preferred prompts.\n\n\u003e - Learn a language model for automatic prompt optimization via reinforcement learning.\n\n![image](https://user-images.githubusercontent.com/1070872/207856962-02f08d92-f2bf-441a-b1c3-efff1a4b6187.png)\n\n\n### Structured Prompting: consume long-sequence prompts in an efficient way\n\n[Paper] [Structured Prompting: Scaling In-Context Learning to 1,000 Examples](https://arxiv.org/abs/2212.06713)\n\n- Example use cases:\n\n\u003e 1) Prepend (many) retrieved (long) documents as context in GPT.\n\n\u003e 2) Scale in-context learning to many demonstration examples.\n\n![image](https://user-images.githubusercontent.com/1070872/207856629-2bb0c933-c27b-4177-9e10-e397622ae79b.png)\n\n\n### X-Prompt: extensible prompts beyond NL for descriptive instructions\n\n[Paper] [Extensible Prompts for Language Models](https://arxiv.org/abs/2212.00616)\n\n\u003e - Extensible interface allowing prompting LLMs beyond natural language for fine-grain specifications\n\n\u003e - Context-guided imaginary word learning for general usability\n\n![Extensible Prompts for Language Models](https://user-images.githubusercontent.com/1070872/207856788-5409d04d-c406-4b29-ae7b-2732e727d4cc.png)\n\n\n## LLMA: LLM Accelerators\n\n### Accelerate LLM Inference with References\n\n[Paper] [Inference with Reference: Lossless Acceleration of Large Language Models](https://arxiv.org/abs/2304.04487)\n\n\u003e - Outputs of LLMs often have significant overlaps with some references (e.g., retrieved documents).\n\n\u003e - LLMA losslessly accelerate the inference of LLMs by copying and verifying text spans from references into the LLM inputs.\n\n\u003e - Applicable to important LLM scenarios such as retrieval-augmented generation and multi-turn conversations.\n\n\u003e - Achieves 2~3 times speed-up without additional models.\n\n![image](https://user-images.githubusercontent.com/6700539/231664563-aec35679-b4ab-4b6b-b6b4-b2b4ea1aab53.png)\n\n\n## Fundamental Understanding of LLMs\n\n### Understanding In-Context Learning\n\n[Paper] [Why Can GPT Learn In-Context? Language Models Secretly Perform Finetuning as Meta Optimizers](https://arxiv.org/abs/2212.10559)\n\n\u003e - According to the demonstration examples, GPT produces meta gradients for In-Context Learning (ICL) through forward computation. ICL works by applying these meta gradients to the model through attention.\n\n\u003e - The meta optimization process of ICL shares a dual view with finetuning that explicitly updates the model parameters with back-propagated gradients.\n\n\u003e - We can translate optimization algorithms (such as SGD with Momentum) to their corresponding Transformer architectures.\n\n![image](https://user-images.githubusercontent.com/1070872/208835096-54407f5f-d136-4747-9629-3219988df5d4.png)\n\n## Hiring: [aka.ms/GeneralAI](https://aka.ms/GeneralAI)\nWe are hiring at all levels (including FTE researchers and interns)! If you are interested in working with us on Foundation Models (aka large-scale pre-trained models) and AGI, NLP, MT, Speech, Document AI and Multimodal AI, please send your resume to \u003ca href=\"mailto:fuwei@microsoft.com\" class=\"x-hidden-focus\"\u003efuwei@microsoft.com\u003c/a\u003e.\n\n## License\nThis project is licensed under the license found in the LICENSE file in the root directory of this source tree.\n\n[Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct)\n\n### Contact Information\n\nFor help or issues using the pre-trained models, please submit a GitHub issue.\nFor other communications, please contact [Furu Wei](http://gitnlp.org/) (`fuwei@microsoft.com`).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrosoft%2Flmops","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmicrosoft%2Flmops","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrosoft%2Flmops/lists"}