Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

Awesome-LLM?tab=readme-ov-file

Awesome-LLM: a curated list of Large Language Model
https://github.com/Hannibal046/Awesome-LLM?tab=readme-ov-file

Last synced: 5 days ago
JSON representation

Open LLM -->
- RLHF LLM
  - BLOOM - BigScience Large Open-science Open-access Multilingual Language Model [BLOOM-LoRA](https://github.com/linhduongtuan/BLOOM-LORA)
  - Mistral - Mistral-7B-v0.1 is a small, yet powerful model adaptable to many use-cases including code and 8k sequence length. Apache 2.0 licence.
  - LLaMA - 2](https://ai.meta.com/llama/) - A foundational large language model. [LLaMA.cpp](https://github.com/ggerganov/llama.cpp) [Lit-LLaMA](https://github.com/Lightning-AI/lit-llama)
  - Alpaca - A model fine-tuned from the LLaMA 7B model on 52K instruction-following demonstrations. [Alpaca.cpp](https://github.com/antimatter15/alpaca.cpp) [Alpaca-LoRA](https://github.com/tloen/alpaca-lora)
  - Vicuna - An Open-Source Chatbot Impressing GPT-4 with 90% ChatGPT Quality.
  - Koala - A Dialogue Model for Academic Research
  - StackLLaMA - A hands-on guide to train LLaMA with RLHF.
  - CaMA - a Chinese-English Bilingual LLaMA Model.
  - Orca - Microsoft's finetuned LLaMA model that reportedly matches GPT3.5, finetuned against 5M of data, ChatGPT, and GPT4
  - StableLM - Stability AI Language Models.
  - YaLM - a GPT-like neural network for generating and processing text. It can be used freely by developers and researchers from all over the world.
  - Dolly - a cheap-to-build LLM that exhibits a surprising degree of the instruction following capabilities exhibited by ChatGPT.
  - Dolly 2.0 - the first open source, instruction-following LLM, fine-tuned on a human-generated instruction dataset licensed for research and commercial use.
  - Cerebras-GPT - A Family of Open, Compute-efficient, Large Language Models.
  - GALACTICA - The GALACTICA models are trained on a large-scale scientific corpus.
  - GALPACA - GALACTICA 30B fine-tuned on the Alpaca dataset.
  - Palmyra - Palmyra Base was primarily pre-trained with English text.
  - Camel - a state-of-the-art instruction-following large language model designed to deliver exceptional performance and versatility.
  - PanGu-α - PanGu-α is a 200B parameter autoregressive pretrained Chinese language model develped by Huawei Noah's Ark Lab, MindSpore Team and Peng Cheng Laboratory.
  - HuggingChat - Powered by Open Assistant's latest model – the best open source chat model right now and @huggingface Inference API.
  - StarCoder - Hugging Face LLM for Code
  - Falcon - Falcon LLM is a foundational large language model (LLM) with 40 billion parameters trained on one trillion tokens. TII has now released Falcon LLM – a 40B model.
  - Baichuan - A series of large language models developed by Baichuan Intelligent Technology.
  - Aquila - 悟道·天鹰语言大模型是首个具备中英双语知识、支持商用许可协议、国内数据合规需求的开源语言大模型。
  - phi-1 - a new large language model for code, with significantly smaller size than competing models.
  - phi-1.5 - a 1.3 billion parameter model trained on a dataset of 30 billion tokens, which achieves common sense reasoning benchmark results comparable to models ten times its size that were trained on datasets more than ten times larger.
  - phi-2 - a 2.7 billion-parameter language model that demonstrates outstanding reasoning and language understanding capabilities, showcasing state-of-the-art performance among base language models with less than 13 billion parameters.
  - Deepseek
  - 7B-base - base](https://modelscope.cn/models/deepseek-ai/deepseek-llm-67b-base/summary),
  - Trending Demo - RWKV-5 trained on 100+ world languages (70% English, 15% multilang, 15% code).
  - MoE-16B-base
  - Qwen series - The large language model series proposed by Alibaba Cloud. ｜阿里云研发的通义千问大模型系列. 包括 [7B](https://huggingface.co/Qwen/Qwen-7B), [72B](https://huggingface.co/Qwen/Qwen-72B), 及各种量化和Chat版本. [Chat Demo](https://huggingface.co/spaces/Qwen/Qwen-72B-Chat-Demo)
  - XVERSE series - Multilingual large language model developed by XVERSE Technology Inc | 由深圳元象科技自主研发的支持多语言的大语言模型. 包括[7B](https://github.com/xverse-ai/XVERSE-7B), [13B](https://github.com/xverse-ai/XVERSE-13B), [65B](https://github.com/xverse-ai/XVERSE-65B)等.
  - Jamba - A Hybrid Transformer-Mamba MoE model, with 52B params, first production grade mamba based LLM, 256K context support.
  - Command-R series - Two multilingual large language models intended for retrieval augmented generation (RAG) and conversational use, at [35](https://huggingface.co/CohereForAI/c4ai-command-r-v01) and [104](https://huggingface.co/CohereForAI/c4ai-command-r-plus) billion parameters. 128k context support.
LLM Tutorials and Courses
- minbpe - Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
- Princeton: Understanding Large Language Models
- State of GPT
- ICML2022-Welcome to the "Big Model" Era: Techniques and Systems to Train and Serve Bigger Models
- Neurips2022-Foundational Robustness of Foundation Models
- Let's build GPT: from scratch, in code, spelled out.
- CS25-Transformers United
- llm-course - Course to get into Large Language Models (LLMs) with roadmaps and Colab notebooks.
- CS324 - Large Language Models
- femtoGPT - Pure Rust implementation of a minimal Generative Pretrained Transformer.
- UWaterloo CS 886 - Recent Advances on Foundation Models.
- ChatGPT Prompt Engineering
Great thoughts about LLM
- Open Pretrained Transformers
- Scaling, emergence, and reasoning in large language models
- How does GPT Obtain its Ability? Tracing Emergent Abilities of Language Models to their Sources
- Why did all of the public reproduction of GPT-3 fail?
- The Next Generation Of Large Language Models
- A Stage Review of Instruction Tuning
- LLM Powered Autonomous Agents
- Why you should work on AI AGENTS!
- Google "We Have No Moat, And Neither Does OpenAI"
- AI competition statement
- Prompt Engineering
- Noam Chomsky: The False Promise of ChatGPT
- Is ChatGPT 175 Billion Parameters? Technical Analysis
- Large Language Model Training in 2023
Tutorials
- RLHF LLM
  - Hung-yi Lee
  - StatQuest
  - Hyung Won Chung
  - Susan Zhang
  - Ameet Deshpande
  - Yao Fu
  - Hung-yi Lee
  - Jay Mody
  - Philipp Schmid
  - HuggingFace
  - HuggingFace
  - 张俊林
  - 大师兄
  - HeptaAI
  - Stephen Wolfram
  - Hung-yi Lee
  - StatQuest
  - StatQuest
  - Jack Cook
  - 康斯坦丁
Milestone Papers
- Attention Is All You Need
- Improving Language Understanding by Generative Pre-Training
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Language Models are Unsupervised Multitask Learners
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- Scaling Laws for Neural Language Models
- Language models are few-shot learners
- Evaluating Large Language Models Trained on Code
- On the Opportunities and Risks of Foundation Models
- Finetuned Language Models are Zero-Shot Learners
- WebGPT: Browser-assisted question-answering with human feedback
- Improving language models by retrieving from trillions of tokens
- Scaling Language Models: Methods, Analysis & Insights from Training Gopher
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Solving Quantitative Reasoning Problems with Language Models
- Training language models to follow instructions with human feedback
- An empirical analysis of compute-optimal large language model training
- OPT: Open Pre-trained Transformer Language Models
- Emergent Abilities of Large Language Models
- Language Models are General-Purpose Interfaces
- GLM-130B: An Open Bilingual Pre-trained Model
- Holistic Evaluation of Language Models
- Galactica: A Large Language Model for Science
- OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning
- LLaMA: Open and Efficient Foundation Language Models
- Language Is Not All You Need: Aligning Perception with Language Models
- PaLM-E: An Embodied Multimodal Language Model
- Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
- Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision
- PaLM 2 Technical Report
- RWKV: Reinventing RNNs for the Transformer Era
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- Improving alignment of dialogue agents via targeted human judgements
- Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
- PaLM: Scaling Language Modeling with Pathways
- LaMDA: Language Models for Dialog Applications
- Mistral 7B
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models
- GPT-4 Technical Report
- Multitask Prompted Training Enables Zero-Shot Task Generalization
- Using Deep and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
- Scaling Instruction-Finetuned Language Models
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
- Attention Is All You Need
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- Scaling Laws for Neural Language Models
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- Evaluating Large Language Models Trained on Code
- On the Opportunities and Risks of Foundation Models
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- LaMDA: Language Models for Dialog Applications
- Using Deep and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
- Training language models to follow instructions with human feedback
- PaLM: Scaling Language Modeling with Pathways
- OPT: Open Pre-trained Transformer Language Models
- Language Models are General-Purpose Interfaces
- Improving alignment of dialogue agents via targeted human judgements
- Scaling Instruction-Finetuned Language Models
- GLM-130B: An Open Bilingual Pre-trained Model
- Holistic Evaluation of Language Models
- BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
- Galactica: A Large Language Model for Science
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Mistral 7B
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- Jamba: A Hybrid Transformer-Mamba Language Model
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- Scaling Language Models: Methods, Analysis & Insights from Training Gopher
- An empirical analysis of compute-optimal large language model training
- Resurrecting Recurrent Neural Networks for Long Sequences
- Visual Instruction Tuning
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
- The Llama 3 Herd of Models
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models
- OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
- Unifying Language Learning Paradigms
- Mistral 7B
Other Papers
- Reasoning using Language Models - Collection of papers and resources on Reasoning using Language Models.
- Awesome LLM Human Preference Datasets - a collection of human preference datasets for LLM instruction tuning, RLHF and evaluation.
- ModelEditingPapers - A paper & resource list on model editing for large language models.
- Awesome-LLM-hallucination - LLM hallucination paper list.
- awesome-hallucination-detection - List of papers on hallucination detection in LLMs.
- LLMsPracticalGuide - A curated list of practical guide resources of LLMs
- Awesome ChatGPT Prompts - A collection of prompt examples to be used with the ChatGPT model.
- awesome-chatgpt-prompts-zh - A Chinese collection of prompt examples to be used with the ChatGPT model.
- Awesome ChatGPT - Curated list of resources for ChatGPT and GPT-3 from OpenAI.
- Chain-of-Thoughts Papers - A trend starts from "Chain of Thought Prompting Elicits Reasoning in Large Language Models.
- Awesome Deliberative Prompting - How to ask LLMs to produce reliable reasoning and make reason-responsive decisions.
- Instruction-Tuning-Papers - A trend starts from `Natrural-Instruction` (ACL 2022), `FLAN` (ICLR 2022) and `T0` (ICLR 2022).
- Chain-of-Thought Hub - Measuring LLMs' Reasoning Performance
- Awesome GPT - A curated list of awesome projects and resources related to GPT, ChatGPT, OpenAI, LLM, and more.
- Awesome GPT-3 - a collection of demos and articles about the [OpenAI GPT-3 API](https://openai.com/blog/openai-api/).
- RWKV-howto - possibly useful materials and tutorial for learning RWKV.
- Awesome LLM Security - A curation of awesome tools, documents and projects about LLM Security.
- Awesome-Align-LLM-Human - A collection of papers and resources about aligning large language models (LLMs) with human.
- Awesome-Code-LLM - An awesome and curated list of best code-LLM for research.
- Awesome-LLM-Compression - Awesome LLM compression research papers and tools.
- Awesome-LLM-Systems - Awesome LLM systems research papers.
- awesome-llm-webapps - A collection of open source, actively maintained web apps for LLM applications.
- Awesome-LLM-Healthcare - The paper list of the review on LLMs in medicine.
- Awesome-LLM-Inference - A curated list of Awesome LLM Inference Paper with codes.
- Awesome-LLM-3D - A curated list of Multi-modal Large Language Model in 3D world, including 3D understanding, reasoning, generation, and embodied agents.
- Awesome-Chinese-LLM - 整理开源的中文大语言模型，以规模较小、可私有化部署、训练成本较低的模型为主，包括底座模型，垂直领域微调及应用，数据集与教程等。
- LLM4Opt - Applying Large language models (LLMs) for diverse optimization tasks (Opt) is an emerging research area. This is a collection of references and papers of LLM4Opt.
- awesome-japanese-llm - 日本語LLMまとめ - Overview of Japanese LLMs.
- awesome-language-model-analysis - This paper list focuses on the theoretical or empirical analysis of language models, e.g., the learning dynamics, expressive capacity, interpretability, generalization, and other interesting topics.
LLM Leaderboard
- Chatbot Arena Leaderboard - a benchmark platform for large language models (LLMs) that features anonymous, randomized battles in a crowdsourced manner.
- AlpacaEval Leaderboard - An Automatic Evaluator for Instruction-following Language Models
- OpenCompass 2.0 LLM Leaderboard - OpenCompass is an LLM evaluation platform, supporting a wide range of models (InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.
- Berkeley Function-Calling Leaderboard - evaluates LLM's ability to call external functions / tools
- Open LLM Leaderboard - aims to track, rank and evaluate LLMs and chatbots as they are released.
- MixEval Leaderboard - a ground-truth-based dynamic benchmark derived from off-the-shelf benchmark mixtures, which evaluates LLMs with a highly capable model ranking (i.e., 0.96 correlation with Chatbot Arena) while running locally and quickly (6% the time and cost of running MMLU).
Open LLM
- - Open Ko-LLM Leaderboard - The Open Ko-LLM Leaderboard objectively evaluates the performance of Korean Large Language Model (LLM).
  - api - 08 | [Paper](https://uploads-ssl.webflow.com/60fd4503684b466578c0d307/61138924626a6981ee09caf6_jurassic_tech_paper.pdf) | - |
  - api - 05 | [Paper](https://arxiv.org/pdf/2205.01068.pdf) | [OPT-175B License Agreement](https://github.com/facebookresearch/metaseq/blob/edefd4a00c24197486a3989abe28ca4eb3881e59/projects/OPT/MODEL_LICENSE.md) |
  - api - 05 | [Paper](https://arxiv.org/pdf/2005.14165.pdf) | - |
  - ckpt - 05 | [Homepage](https://falconllm.tii.ae) | [Apache 2.0](https://huggingface.co/tiiuae) |
  - ckpt - 05 | [Paper](https://arxiv.org/pdf/2205.05131v1.pdf) | [Apache 2.0](https://huggingface.co/google/ul2) |
  - ckpt - 04 | [Paper](https://arxiv.org/pdf/2104.12369.pdf) | [Apache 2.0](https://github.com/huawei-noah/Pretrained-Language-Model/blob/4624dbadfe00e871789b509fe10232c77086d1de/PanGu-%CE%B1/LICENSE) |
  - ckpt - 10 | [Paper](https://jmlr.org/papers/v21/20-074.html) | [Apache 2.0](https://huggingface.co/t5-11b) |
  - api - 10 | [Paper](https://arxiv.org/pdf/2012.00413.pdf) | - |
  - ckpt - 09 | [Github](https://github.com/BlinkDL/RWKV-LM) | [Apache 2.0](https://huggingface.co/BlinkDL/rwkv-4-pile-7b) |
  - Yet Another LLM Leaderboard - Leaderboard made with LLM AutoEval using Nous benchmark suite.
  - RecurrentGemma-2B
  - DeepSeek-Coder-v2-16|236B-MOE
  - Pythia-1|1.4|2.8|6.9|12B
  - Mixtral-8x7B
  - Qwen2-0.5|1.5|7|57-MOE|72B
  - Nemotron-4-340B
  - DBRX-132B-MoE
  - api - 05 | [Paper](https://arxiv.org/pdf/2005.14165.pdf) | - |
  - Llama 3.1-8|70|405B
  - Codestral-7|22B
  - Gemma-2|7B
  - BLOOMZ&mT0
  - Qwen1.5-0.5B|1.8B|4B|7B|14B|32B|72B|110B|MoE-A2.7B
  - Qwen2.5-0.5B|1.5B|3B|7B|14B|32B|72B
  - CodeQwen1.5-7B
  - Qwen2.5-Coder-1.5B|7B|32B
  - Qwen2-Math-1.5B|7B|72B
  - Qwen2.5-Math-1.5B|7B|72B
  - Qwen2-VL-2B|7B|72B
  - Qwen2-Audio-7B
  - Llama 2-7|13|70B
  - OPT-1.3|6.7|13|30|66B
  - Mistral-7B
  - Mixtral-8x22B
  - T5
  - OpenELM-1.1|3B
  - Phi1-1.3B
  - Phi2-2.7B
  - Phi3-3.8|7|14B
  - OLMo-7B
  - Grok-1-314B-MoE
  - Command R-35B
  - DeepSeek-Math-7B
  - DeepSeek-Coder-1.3|6.7|7|33B
  - DeepSeek-VL-1.3|7B
  - DeepSeek-MoE-16B
  - DeepSeek-v2-236B-MoE
  - Qwen-1.8B|7B|14B|72B
  - Qwen1.5-1.8|4|7|14|32|72|110B
  - CodeQwen-7B
  - Qwen-VL-7B
  - Yi-34B
  - Yi1.5-6|9|34B
  - Yi-VL-6B|34B
  - Baichuan2-7|13B
  - GLM-2|6|10|13|70B
  - CogVLM2-19B
  - MiniCPM-2B
  - OmniLLM-12B
  - VisCPM-10B
  - CPM-Bee-1|2|5|10B
  - RWKV-v4|5|6
  - StableLM-v2-1.6|12B
  - StableCode-3B
  - StarCoder-1|3|7B
  - StarCoder2-3|7|15B
  - MPT-7B
  - InternLM2-1.8|7|20B
  - InternLM-Math-7B|20B
  - InternLM-XComposer2-1.8|7B
  - InternVL-2|6|14|26
  - Gemma2-9|27B
- Instruction finetuned LLM
  - api - 03 | [Paper](https://arxiv.org/pdf/2203.02155.pdf) | - |
  - ckpt - 11| [Paper](https://arxiv.org/pdf/2211.09085.pdf)| [CC-BY-NC-4.0](https://github.com/paperswithcode/galai/blob/3a724f562af1a0c8ff97a096c5fbebe579e2160f/LICENSE-MODEL.md) |
  - ckpt - 03 | [Blog](https://www.yitay.net/blog/flan-ul2-20b)| [Apache 2.0](https://huggingface.co/google/flan-ul2) |
  - ckpt - 10|[Paper](https://arxiv.org/pdf/2210.11416.pdf)| [Apache 2.0](https://github.com/google-research/t5x/blob/776279bdacd8c5a2d3e8ce0f2e7064bd98e98b47/LICENSE) |
  - ckpt - 10|[Paper](https://arxiv.org/pdf/2110.08207.pdf)| [Apache 2.0](https://huggingface.co/bigscience/T0) |
  - demo - 03|[Github](https://github.com/tatsu-lab/stanford_alpaca)| [CC BY NC 4.0](https://github.com/tatsu-lab/stanford_alpaca/blob/main/WEIGHT_DIFF_LICENSE) |
  - ckpt - 06|[Paper](https://arxiv.org/pdf/2306.02707)|[Non-commercial bespoke license](https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md) |
  - ckpt - 06|[Paper](https://arxiv.org/pdf/2306.02707)|[Non-commercial bespoke license](https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md) |
- RLHF LLM
  - demo - 03 | [Blog](https://www.anthropic.com/index/introducing-claude) |
  - survey paper
  - demo - x9IvKno0A4sk30) | 2022-11 | [Blog](https://openai.com/blog/chatgpt/) |
Courses
- RLHF LLM
  - OpenBMB
  - Stanford
  - Stanford Webinar
  - 李沐
  - 陳縕儂
  - 李沐
  - 李沐 - 7UM8iUTj3qKqdhbQULP5I&index=18)
  - Aston Zhang - 7UM8iUTj3qKqdhbQULP5I&index=29)
  - MIT
  - DeepLearning.AI
  - DeepLearning.AI
  - DeepLearning.AI
  - DeepLearning.AI
  - DeepLearning.AI
  - DeepLearning.AI
  - DeepLearning.AI
  - DeepLearning.AI
  - DeepLearning.AI
  - DeepLearning.AI
  - DeepLearning.AI
  - DeepLearning.AI
LLM Training Frameworks
- RLHF LLM
  - FairScale - FairScale is a PyTorch extension library for high performance and large scale training.
- - DeepSpeed - DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
  - Megatron-DeepSpeed - DeepSpeed version of NVIDIA's Megatron-LM that adds additional support for several features such as MoE model training, Curriculum Learning, 3D Parallelism, and others.
  - torchtune - A Native-PyTorch Library for LLM Fine-tuning.
  - torchtitan - A native PyTorch Library for large model training.
  - Megatron-LM - Ongoing research training transformer models at scale.
  - Colossal-AI - Making large AI models cheaper, faster, and more accessible.
  - BMTrain - Efficient Training for Big Models.
  - Mesh Tensorflow - Mesh TensorFlow: Model Parallelism Made Easier.
  - maxtext - A simple, performant and scalable Jax LLM!
  - GPT-NeoX - An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library.
  - Alpa - Alpa is a system for training and serving large-scale neural networks.
Deploying Tools
- RLHF LLM
  - Sidekick - Data integration platform for LLMs.
  - LiteChain - Lightweight alternative to LangChain for composing LLMs
  - promptfoo - Test your prompts. Evaluate and compare LLM outputs, catch regressions, and improve prompt quality.
Prompting libraries & tools
- RLHF LLM
  - LangChain
  - Guidance
  - Chainlit
  - Guardrails.ai
  - Outlines - specific language to simplify prompting and constrain generation.
  - Scale Spellbook
  - Weights & Biases
  - LlamaIndex
  - LMQL
LLM Applications
- FLAML (A Fast Library for Automated Machine Learning & Tuning)
- PromptPerfect
- Arthur Shield
- GPTRouter - GPTRouter is an open source LLM API Gateway that offers a universal API for 30+ LLMs, vision, and image models, with smart fallbacks based on uptime and latency, automatic retries, and streaming. Stay operational even when OpenAI is down
- LangChain
- AdalFlow - AdalFlow: The PyTorch library for LLM applications.
- YiVal - source GenAI-Ops tool for tuning and evaluating prompts, configurations, and model parameters using customizable datasets, evaluation methods, and improvement strategies.
- Guidance
- Evidently - source framework to evaluate, test and monitor ML and LLM-powered systems.
- Semantic Kernel
- Prompttools - source Python tools for testing and evaluating models, vector DBs, and prompts.
- Promptify
- OpenAI Evals - source library for evaluating task performance of language models and prompts.
- LlamaIndex
- Flappy - Ready LLM Agent SDK for Every Developer.
- QAnything - A local knowledge base question-answering system designed to support a wide range of file formats and databases.
- llm-ui - A React library for building LLM UIs.
- Dify - An open-source LLM app development platform with an intuitive interface that streamlines AI workflows, model management, and production deployment.
- LazyLLM - An open-source LLM app for building multi-agent LLMs applications in an easy and lazy way, supports model deployment and fine-tuning.
- MemFree - Open Source Hybrid AI Search Engine, Instantly Get Accurate Answers from the Internet, Bookmarks, Notes, and Docs. Support One-Click Deployment
- ModelFusion - A TypeScript library for building apps with LLMs and other ML models (speech-to-text, text-to-speech, image generation).
- unslothai - A framework that specializes in efficient fine-tuning. On its GitHub page, you can find ready-to-use fine-tuning templates for various LLMs, allowing you to easily train your own data for free on the Google Colab cloud.
- Outlines - specific language to simplify prompting and constrain generation.
- Weights & Biases
- dspy - DSPy: The framework for programming—not prompting—foundation models.
- OneKE - English knowledge extraction model with knowledge graphs and natural language processing technologies.
- Wallaroo.AI - Deploy, manage, optimize any model at scale across any environment from cloud to edge. Let's you go from python notebook to inferencing in minutes.
Books
- RLHF LLM
  - Generative AI with LangChain: Build large language model (LLM) apps with Python, ChatGPT, and other LLMs - it comes with a [GitHub repository](https://github.com/benman1/generative_ai_with_langchain) that showcases a lot of the functionality
Opinions
- RLHF LLM
  - 我的大模型世界观 - 04-23] [陆奇]
  - Towards ChatGPT and Beyond - 02-20][知乎][欧泽彬]
  - 追赶ChatGPT的难点与平替 - 02-19][李rumor]
  - 对话旷视研究院张祥雨｜ChatGPT的科研价值可能更大 - 02-16][知乎][旷视科技]
  - 关于ChatGPT八个技术问题的猜想 - 02-15][知乎][张家俊]
  - ChatGPT发展历程、原理、技术架构详解和产业未来 - 02-15][知乎][陈巍谈芯]
  - 对ChatGPT的二十点看法 - 02-13]\[知乎][熊德意]
  - ChatGPT-所见、所闻、所感 - 02-11]\[知乎][刘聪NLP]
  - Large Language Models: A New Moore's Law - 10-26\]\[Huggingface\]
Other Useful Resources
- RLHF LLM
  - Emergent Mind - The latest AI news, curated & explained by GPT-4.
  - ShareGPT - Share your wildest ChatGPT conversations with one click.
  - Cohere Summarize Beta - Introducing Cohere Summarize Beta: A New Endpoint for Text Summarization
  - chatgpt-wrapper - ChatGPT Wrapper is an open-source unofficial Python API and CLI that lets you interact with ChatGPT.
  - Cursor - Write, edit, and chat about your code with a powerful AI.
  - AutoGPT - an experimental open-source application showcasing the capabilities of the GPT-4 language model.
  - MTEB - Massive Text Embedding Benchmark Leaderboard
Trending LLM Projects
- LibreChat - All-In-One AI Conversations with LibreChat.
- Open-Sora - Democratizing Efficient Video Production for All.
- LLM101n - Let's build a Storyteller.
- miqu-1-70b - A leaked 70B model from Mistral AI.
- Deep-Live-Cam - real time face swap and one-click video deepfake with only a single image (uncensored).
- MiniCPM-V 2.6 - A GPT-4V Level MLLM for Single Image, Multi Image and Video on Your Phone
- GPT-SoVITS - 1 min voice data can also be used to train a good TTS model! (few shot voice cloning).
- Sora - Sora is an AI model that can create realistic and imaginative scenes from text instructions.
- GPT-4o - OpenAI's new flagship model that can reason across audio, vision, and text in real time.
LLM Evaluation:
- lm-evaluation-harness - A framework for few-shot evaluation of language models.
- MixEval - A reliable click-and-go evaluation suite compatible with both open-source and proprietary models, supporting MixEval and other benchmarks.
- lighteval - a lightweight LLM evaluation suite that Hugging Face has been using internally.
- OLMO-eval - a repository for evaluating open language models.
- instruct-eval - This repository contains code to quantitatively evaluate instruction-tuned models such as Alpaca and Flan-T5 on held-out tasks.
- simple-evals - Eval tools by OpenAI.
- Giskard - Testing & evaluation library for LLM applications, in particular RAGs
- Ragas - a framework that helps you evaluate your Retrieval Augmented Generation (RAG) pipelines.
- LangSmith - a unified platform from LangChain framework for: evaluation, collaboration HITL (Human In The Loop), logging and monitoring LLM applications.
LLM Deployment
- llm-inference-solutions
- vLLM - A high-throughput and memory-efficient inference and serving engine for LLMs.
- exllama - A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.
- llama.cpp - LLM inference in C/C++.
- ollama - Get up and running with Llama 3, Mistral, Gemma, and other large language models.
- FastChat - A distributed multi-model LLM serving system with web UI and OpenAI-compatible RESTful APIs.
- mistral.rs - Blazingly fast LLM inference.
- MindSQL - A python package for Txt-to-SQL with self hosting functionalities and RESTful APIs compatible with proprietary as well as open source LLM.
- SkyPilot - Run LLMs and batch jobs on any cloud. Get maximum cost savings, highest GPU availability, and managed execution -- all with a simple interface.
- QA-Pilot - An interactive chat project that leverages Ollama/OpenAI/MistralAI LLMs for rapid understanding and navigation of GitHub code repository or compressed file resources.
- Shell-Pilot - Interact with LLM using Ollama models(or openAI, mistralAI)via pure shell scripts on your Linux(or MacOS) system, enhancing intelligent system management without any dependencies.
- Floom
- Swiss Army Llama - Comprehensive set of tools for working with local LLMs for various tasks.
- magentic - Seamlessly integrate LLMs as Python functions
- wechat-chatgpt - Use ChatGPT On Wechat via wechaty
- Agenta - Easily build, version, evaluate and deploy your LLM-powered apps.
- Serge - a chat interface crafted with llama.cpp for running Alpaca models. No API keys, entirely self-hosted!
- Langroid - Harness LLMs with Multi-Agent Programming
- CometLLM - A 100% opensource LLMOps platform to log, manage, and visualize your LLM prompts and chains. Track prompt templates, prompt variables, prompt duration, token usage, and other metadata. Score prompt outputs and visualize chat history all within a single UI.
- IntelliServer - simplifies the evaluation of LLMs by providing a unified microservice to access and test multiple AI models.
- OpenLLM - Fine-tune, serve, deploy, and monitor any open-source LLMs in production. Used in production at [BentoML](https://bentoml.com/) for LLMs-based applications.
- DeepSpeed-Mii - MII makes low-latency and high-throughput inference, similar to vLLM powered by DeepSpeed.
- Text-Embeddings-Inference - Inference for text-embeddings in Rust, HFOIL Licence.
- Infinity - Inference for text-embeddings in Python
- TensorRT-LLM - Nvidia Framework for LLM Inference
- FasterTransformer - NVIDIA Framework for LLM Inference(Transitioned to TensorRT-LLM)
- Flash-Attention - A method designed to enhance the efficiency of Transformer models
- Langchain-Chatchat - Formerly langchain-ChatGLM, local knowledge based LLM (like ChatGLM) QA app with langchain.
- Search with Lepton - Build your own conversational search engine using less than 500 lines of code by [LeptonAI](https://github.com/leptonai).
- Robocorp - Create, deploy and operate Actions using Python anywhere to enhance your AI agents and assistants. Batteries included with an extensive set of libraries, helpers and logging.
- LMDeploy - A high-throughput and low-latency inference and serving framework for LLMs and VLs
- LLocalSearch - Locally running websearch using LLM chains
- AI Gateway - ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency.
- talkd.ai dialog - Simple API for deploying any RAG or LLM that you want adding plugins.
- Wllama - WebAssembly binding for llama.cpp - Enabling in-browser LLM inference
- Embedchain - Framework to create ChatGPT like bots over your dataset.
- promptfoo - Test your prompts. Evaluate and compare LLM outputs, catch regressions, and improve prompt quality.
- GPUStack - An open-source GPU cluster manager for running LLMs
- SGLang - SGLang is a fast serving framework for large language models and vision language models.
- Langfuse - Open Source LLM Engineering Platform 🪢 Tracing, Evaluations, Prompt Management, Evaluations and Playground.
- TGI - a toolkit for deploying and serving Large Language Models (LLMs).
- Haystack - an open-source NLP framework that allows you to use LLMs and transformer-based models from Hugging Face, OpenAI and Cohere to interact with your own data.
Miscellaneous
- Open-evals - A framework extend openai's [Evals](https://github.com/openai/evals) for different language model.
- OpenAGI - When LLM Meets Domain Experts.
- EasyEdit - An easy-to-use framework to edit large language models.
- chatgpt-shroud - A Chrome extension for OpenAI's ChatGPT, enhancing user privacy by enabling easy hiding and unhiding of chat history. Ideal for privacy during screen shares.
- AutoGPT - an experimental open-source application showcasing the capabilities of the GPT-4 language model.
- chatgpt-wrapper - ChatGPT Wrapper is an open-source unofficial Python API and CLI that lets you interact with ChatGPT.
- Arize-Phoenix - Open-source tool for ML observability that runs in your notebook environment. Monitor and fine tune LLM, CV and Tabular Models.
- Major LLMs + Data Availability
- 500+ Best AI Tools
LLM Books
- Build a Large Language Model (From Scratch) - A guide to building your own working LLM.
- Generative AI with LangChain: Build large language model (LLM) apps with Python, ChatGPT, and other LLMs - it comes with a [GitHub repository](https://github.com/benman1/generative_ai_with_langchain) that showcases a lot of the functionality
- Generative AI with LangChain: Build large language model (LLM) apps with Python, ChatGPT, and other LLMs - it comes with a [GitHub repository](https://github.com/benman1/generative_ai_with_langchain) that showcases a lot of the functionality
- BUILD GPT: HOW AI WORKS - explains how to code a Generative Pre-trained Transformer, or GPT, from scratch.
LLM Data
- LLMDataHub
- IBM data-prep-kit - Open-Source Toolkit for Efficient Unstructured Data Processing with Pre-built Modules and Local to Cluster Scalability.

Programming Languages

Python 62 Jupyter Notebook 9 TypeScript 9 Rust 4 C++ 4 JavaScript 2 C# 2 Go 2 Svelte 1 Shell 1

Ecosyste.ms: Awesome

Awesome-LLM?tab=readme-ov-file

Open LLM -->

RLHF LLM

LLM Tutorials and Courses

Great thoughts about LLM

Tutorials

RLHF LLM

Milestone Papers

Other Papers

LLM Leaderboard

Open LLM

Instruction finetuned LLM

RLHF LLM

Courses

RLHF LLM

LLM Training Frameworks

RLHF LLM

Deploying Tools

RLHF LLM

Prompting libraries & tools

RLHF LLM

LLM Applications

Books

RLHF LLM

Opinions

RLHF LLM

Other Useful Resources

RLHF LLM

Trending LLM Projects

LLM Evaluation:

LLM Deployment

Miscellaneous

LLM Books

LLM Data