Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/microsoft/lmops
General technology for enabling AI capabilities w/ LLMs and MLLMs
https://github.com/microsoft/lmops
agi gpt language-model llm lm lmops nlp pretraining prompt promptist x-prompt
Last synced: about 1 month ago
JSON representation
General technology for enabling AI capabilities w/ LLMs and MLLMs
- Host: GitHub
- URL: https://github.com/microsoft/lmops
- Owner: microsoft
- License: mit
- Created: 2022-12-13T12:56:21.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-07-30T21:44:17.000Z (about 1 month ago)
- Last Synced: 2024-07-31T05:25:53.265Z (about 1 month ago)
- Topics: agi, gpt, language-model, llm, lm, lmops, nlp, pretraining, prompt, promptist, x-prompt
- Language: Python
- Homepage: https://aka.ms/GeneralAI
- Size: 69.3 MB
- Stars: 3,463
- Watchers: 56
- Forks: 258
- Open Issues: 35
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md
- Support: SUPPORT.md
Awesome Lists containing this project
- awesome-llm - LMOps - Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities. (Models / Projects)
- awesome-ChatGPT-repositories - LMOps - General technology for enabling AI capabilities w/ LLMs and MLLMs (Prompts)
README
# LMOps
LMOps is a research initiative on fundamental research and technology for building AI products w/ foundation models, especially on the general technology for enabling AI capabilities w/ LLMs and Generative AI models.
- Better Prompts: [Automatic Prompt Optimization](https://arxiv.org/abs/2305.03495), [Promptist](https://arxiv.org/abs/2212.09611), [Extensible prompts](https://arxiv.org/abs/2212.00616), [Universal prompt retrieval](https://arxiv.org/abs/2303.08518), [LLM Retriever](https://arxiv.org/abs/2307.07164), [In-Context Demonstration Selection](https://arxiv.org/abs/2305.14726)
- Longer Context: [Structured prompting](https://arxiv.org/abs/2212.06713), [Length-Extrapolatable Transformers](https://arxiv.org/abs/2212.10554)
- LLM Alignment: [Alignment via LLM feedback]()
- LLM Accelerator (Faster Inference): [Lossless Acceleration of LLMs](https://arxiv.org/abs/2304.04487)
- LLM Customization: [Adapt LLM to domains](https://arxiv.org/pdf/2309.09530.pdf)
- Fundamentals: [Understanding In-Context Learning](https://arxiv.org/abs/2212.10559)
## Links
- [microsoft/unilm](https://github.com/microsoft/unilm): Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
- [microsoft/torchscale](https://github.com/microsoft/torchscale): Transformers at (any) Scale
## News
- [Paper Release] Nov, 2023: [In-Context Demonstration Selection with Cross Entropy Difference](https://arxiv.org/abs/2305.14726) (EMNLP 2023)
- [Paper Release] Oct, 2023: [Tuna: Instruction Tuning using Feedback from Large Language Models](https://arxiv.org/pdf/2310.13385.pdf) (EMNLP 2023)
- [Paper Release] Oct, 2023: [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://arxiv.org/abs/2305.03495) (EMNLP 2023)
- [Paper Release] Oct, 2023: [UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation](https://arxiv.org/abs/2303.08518) (EMNLP 2023)
- [Paper Release] July, 2023: [Learning to Retrieve In-Context Examples for Large Language Models](https://arxiv.org/abs/2307.07164)
- [Paper Release] April, 2023: [Inference with Reference: Lossless Acceleration of Large Language Models](https://arxiv.org/abs/2304.04487)
- [Paper Release] Dec, 2022: [Why Can GPT Learn In-Context? Language Models Secretly Perform Finetuning as Meta Optimizers](https://arxiv.org/abs/2212.10559)
- [Paper & Model & Demo Release] Dec, 2022: [Optimizing Prompts for Text-to-Image Generation](https://aka.ms/promptist)
- [Paper & Code Release] Dec, 2022: [Structured Prompting: Scaling In-Context Learning to 1,000 Examples](https://arxiv.org/abs/2212.06713)
- [Paper Release] Nov, 2022: [Extensible Prompts for Language Models](https://arxiv.org/abs/2212.00616)
## Prompt Intelligence
Advanced technologies facilitating prompting language models.
### Promptist: reinforcement learning for automatic prompt optimization
[Paper] [Optimizing Prompts for Text-to-Image Generation](https://arxiv.org/abs/2212.09611)
> - Language models serve as a prompt interface that optimizes user input into model-preferred prompts.
> - Learn a language model for automatic prompt optimization via reinforcement learning.
### Structured Prompting: consume long-sequence prompts in an efficient way
[Paper] [Structured Prompting: Scaling In-Context Learning to 1,000 Examples](https://arxiv.org/abs/2212.06713)
- Example use cases:
> 1) Prepend (many) retrieved (long) documents as context in GPT.
> 2) Scale in-context learning to many demonstration examples.
### X-Prompt: extensible prompts beyond NL for descriptive instructions
[Paper] [Extensible Prompts for Language Models](https://arxiv.org/abs/2212.00616)
> - Extensible interface allowing prompting LLMs beyond natural language for fine-grain specifications
> - Context-guided imaginary word learning for general usability
## LLMA: LLM Accelerators
### Accelerate LLM Inference with References
[Paper] [Inference with Reference: Lossless Acceleration of Large Language Models](https://arxiv.org/abs/2304.04487)
> - Outputs of LLMs often have significant overlaps with some references (e.g., retrieved documents).
> - LLMA losslessly accelerate the inference of LLMs by copying and verifying text spans from references into the LLM inputs.
> - Applicable to important LLM scenarios such as retrieval-augmented generation and multi-turn conversations.
> - Achieves 2~3 times speed-up without additional models.
## Fundamental Understanding of LLMs
### Understanding In-Context Learning
[Paper] [Why Can GPT Learn In-Context? Language Models Secretly Perform Finetuning as Meta Optimizers](https://arxiv.org/abs/2212.10559)
> - According to the demonstration examples, GPT produces meta gradients for In-Context Learning (ICL) through forward computation. ICL works by applying these meta gradients to the model through attention.
> - The meta optimization process of ICL shares a dual view with finetuning that explicitly updates the model parameters with back-propagated gradients.
> - We can translate optimization algorithms (such as SGD with Momentum) to their corresponding Transformer architectures.
## Hiring: [aka.ms/GeneralAI](https://aka.ms/GeneralAI)
We are hiring at all levels (including FTE researchers and interns)! If you are interested in working with us on Foundation Models (aka large-scale pre-trained models) and AGI, NLP, MT, Speech, Document AI and Multimodal AI, please send your resume to [email protected].
## License
This project is licensed under the license found in the LICENSE file in the root directory of this source tree.
[Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct)
### Contact Information
For help or issues using the pre-trained models, please submit a GitHub issue.
For other communications, please contact [Furu Wei](http://gitnlp.org/) (`[email protected]`).