Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

Awesome-LLM

Awesome-LLM: a curated list of Large Language Model
https://github.com/Hannibal046/Awesome-LLM

Last synced: 8 days ago
JSON representation

  • Books

  • Courses

  • Deploying Tools

    • RLHF LLM

      • Tune Studio - Playground for devs to finetune & deploy LLMs

  • Great thoughts about LLM

  • LLM Applications

    • FLAML (A Fast Library for Automated Machine Learning & Tuning)
    • PromptPerfect
    • Arthur Shield
    • GPTRouter - GPTRouter is an open source LLM API Gateway that offers a universal API for 30+ LLMs, vision, and image models, with smart fallbacks based on uptime and latency, automatic retries, and streaming. Stay operational even when OpenAI is down
    • OneKE - English knowledge extraction model with knowledge graphs and natural language processing technologies.
    • Wallaroo.AI - Deploy, manage, optimize any model at scale across any environment from cloud to edge. Let's you go from python notebook to inferencing in minutes.
    • Weights & Biases
    • AdalFlow - AdalFlow: The PyTorch library for LLM applications.
    • dspy - DSPy: The framework for programming—not prompting—foundation models.
    • YiVal - source GenAI-Ops tool for tuning and evaluating prompts, configurations, and model parameters using customizable datasets, evaluation methods, and improvement strategies.
    • Evidently - source framework to evaluate, test and monitor ML and LLM-powered systems.
    • Semantic Kernel
    • Prompttools - source Python tools for testing and evaluating models, vector DBs, and prompts.
    • Promptify
    • OpenAI Evals - source library for evaluating task performance of language models and prompts.
    • Flappy - Ready LLM Agent SDK for Every Developer.
    • QAnything - A local knowledge base question-answering system designed to support a wide range of file formats and databases.
    • llm-ui - A React library for building LLM UIs.
    • Dify - An open-source LLM app development platform with an intuitive interface that streamlines AI workflows, model management, and production deployment.
    • LazyLLM - An open-source LLM app for building multi-agent LLMs applications in an easy and lazy way, supports model deployment and fine-tuning.
    • MemFree - Open Source Hybrid AI Search Engine, Instantly Get Accurate Answers from the Internet, Bookmarks, Notes, and Docs. Support One-Click Deployment
    • unslothai - A framework that specializes in efficient fine-tuning. On its GitHub page, you can find ready-to-use fine-tuning templates for various LLMs, allowing you to easily train your own data for free on the Google Colab cloud.
    • awesome-llm-apps
    • LangChain
    • LlamaIndex
    • MLflow - MLflow: An open-source framework for the end-to-end machine learning lifecycle, helping developers track experiments, evaluate models/prompts, deploy models, and add observability with tracing.
    • Embedchain - Framework to create ChatGPT like bots over your dataset.
    • Tune Studio - Playground for devs to finetune & deploy LLMs
    • MNN-LLM - - A Device-Inference framework, including LLM Inference on device(Mobile Phone/PC/IOT)
    • CAMEL - First LLM Multi-agent framework.
    • Guidance
    • Chainlit
    • Outlines - specific language to simplify prompting and constrain generation.
    • Weights & Biases
    • AutoRAG - Open source AutoML tool for RAG. Optimize the RAG answer quality automatically. From generation evaluation datset to deploying optimized RAG pipeline.
    • Epsilla - An all-in-one LLM Agent platform with your private data and knowledge, delivers your production-ready AI Agents on Day 1.
    • Just-Chat - Make your LLM agent and chat with it simple and fast!
    • Agentic Radar - Open-source CLI security scanner for agentic workflows. Scans your workflow’s source code, detects vulnerabilities, and generates an interactive visualization along with a detailed security report. Supports LangGraph, CrewAI, n8n, OpenAI Agents, and more.
    • LangWatch - Open-source LLM observability, prompt evaulation, and prompt optimzation platform.
    • TensorZero - TensorZero is an open-source framework for building production-grade LLM applications. It unifies an LLM gateway, observability, optimization, evaluations, and experimentation.
    • LiteChain - Lightweight alternative to LangChain for composing LLMs

  • LLM Books

  • LLM Data

    • IBM data-prep-kit - Open-Source Toolkit for Efficient Unstructured Data Processing with Pre-built Modules and Local to Cluster Scalability.
    • Datatrove - Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
    • Dingo - Dingo: A Comprehensive Data Quality Evaluation Tool
    • FastDatasets - A powerful tool for creating high-quality training datasets for Large Language Models
    • IBM data-prep-kit - Open-Source Toolkit for Efficient Unstructured Data Processing with Pre-built Modules and Local to Cluster Scalability.

  • LLM Deployment

    • Haystack - an open-source NLP framework that allows you to use LLMs and transformer-based models from Hugging Face, OpenAI and Cohere to interact with your own data.
    • TGI - a toolkit for deploying and serving Large Language Models (LLMs).
    • llm-inference-solutions
    • vLLM - A high-throughput and memory-efficient inference and serving engine for LLMs.
    • exllama - A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.
    • ollama - Get up and running with Llama 3, Mistral, Gemma, and other large language models.
    • Langfuse - Open Source LLM Engineering Platform 🪢 Tracing, Evaluations, Prompt Management, Evaluations and Playground.
    • FastChat - A distributed multi-model LLM serving system with web UI and OpenAI-compatible RESTful APIs.
    • mistral.rs - Blazingly fast LLM inference.
    • MindSQL - A python package for Txt-to-SQL with self hosting functionalities and RESTful APIs compatible with proprietary as well as open source LLM.
    • SkyPilot - Run LLMs and batch jobs on any cloud. Get maximum cost savings, highest GPU availability, and managed execution -- all with a simple interface.
    • QA-Pilot - An interactive chat project that leverages Ollama/OpenAI/MistralAI LLMs for rapid understanding and navigation of GitHub code repository or compressed file resources.
    • Shell-Pilot - Interact with LLM using Ollama models(or openAI, mistralAI)via pure shell scripts on your Linux(or MacOS) system, enhancing intelligent system management without any dependencies.
    • Floom
    • Swiss Army Llama - Comprehensive set of tools for working with local LLMs for various tasks.
    • magentic - Seamlessly integrate LLMs as Python functions
    • wechat-chatgpt - Use ChatGPT On Wechat via wechaty
    • Agenta - Easily build, version, evaluate and deploy your LLM-powered apps.
    • Serge - a chat interface crafted with llama.cpp for running Alpaca models. No API keys, entirely self-hosted!
    • Langroid - Harness LLMs with Multi-Agent Programming
    • IntelliServer - simplifies the evaluation of LLMs by providing a unified microservice to access and test multiple AI models.
    • OpenLLM - Fine-tune, serve, deploy, and monitor any open-source LLMs in production. Used in production at [BentoML](https://bentoml.com/) for LLMs-based applications.
    • Text-Embeddings-Inference - Inference for text-embeddings in Rust, HFOIL Licence.
    • Infinity - Inference for text-embeddings in Python
    • TensorRT-LLM - Nvidia Framework for LLM Inference
    • FasterTransformer - NVIDIA Framework for LLM Inference(Transitioned to TensorRT-LLM)
    • Flash-Attention - A method designed to enhance the efficiency of Transformer models
    • Langchain-Chatchat - Formerly langchain-ChatGLM, local knowledge based LLM (like ChatGLM) QA app with langchain.
    • Search with Lepton - Build your own conversational search engine using less than 500 lines of code by [LeptonAI](https://github.com/leptonai).
    • Robocorp - Create, deploy and operate Actions using Python anywhere to enhance your AI agents and assistants. Batteries included with an extensive set of libraries, helpers and logging.
    • LMDeploy - A high-throughput and low-latency inference and serving framework for LLMs and VLs
    • LLocalSearch - Locally running websearch using LLM chains
    • AI Gateway - ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency.
    • talkd.ai dialog - Simple API for deploying any RAG or LLM that you want adding plugins.
    • Wllama - WebAssembly binding for llama.cpp - Enabling in-browser LLM inference
    • GPUStack - An open-source GPU cluster manager for running LLMs
    • SGLang - SGLang is a fast serving framework for large language models and vision language models.
    • Opik - Confidently evaluate, test, and ship LLM applications with a suite of observability tools to calibrate language model outputs across your dev and production lifecycle.

  • LLM Evaluation:

    • LangSmith - a unified platform from LangChain framework for: evaluation, collaboration HITL (Human In The Loop), logging and monitoring LLM applications.
    • lm-evaluation-harness - A framework for few-shot evaluation of language models.
    • lighteval - a lightweight LLM evaluation suite that Hugging Face has been using internally.
    • OLMO-eval - a repository for evaluating open language models.
    • instruct-eval - This repository contains code to quantitatively evaluate instruction-tuned models such as Alpaca and Flan-T5 on held-out tasks.
    • simple-evals - Eval tools by OpenAI.
    • Giskard - Testing & evaluation library for LLM applications, in particular RAGs
    • Ragas - a framework that helps you evaluate your Retrieval Augmented Generation (RAG) pipelines.
    • HELM - Holistic Evaluation of Language Models (HELM), a framework to increase the transparency of language models.

  • LLM Inference

    • MInference - To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.
    • DeepSpeed-Mii - MII makes low-latency and high-throughput inference, similar to vLLM powered by DeepSpeed.
    • Liger-Kernel - Efficient Triton Kernels for LLM Training.
    • prima.cpp - A distributed implementation of llama.cpp that lets you run 70B-level LLMs on your everyday devices.
    • deploy-llms-with-ansible - Easily deploy any LLM on a VM with minimal configuration, using Ansible.
    • llama.cpp - LLM inference in C/C++.

  • LLM Leaderboard

    • Open LLM Leaderboard - aims to track, rank, and evaluate LLMs and chatbots as they are released.
    • OpenCompass 2.0 LLM Leaderboard - OpenCompass is an LLM evaluation platform, supporting a wide range of models (InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.
    • Berkeley Function-Calling Leaderboard - evaluates LLM's ability to call external functions/tools.
    • MixEval - a ground-truth-based dynamic benchmark derived from off-the-shelf benchmark mixtures, which evaluates LLMs with a highly capable model ranking (i.e., 0.96 correlation with Chatbot Arena) while running locally and quickly (6% the time and cost of running MMLU).
    • AlpacaEval - An Automatic Evaluator for Instruction-following Language Models using Nous benchmark suite.
    • ACLUE - an evaluation benchmark focused on ancient Chinese language comprehension.
    • BeHonest - A pioneering benchmark specifically designed to assess honesty in LLMs comprehensively.
    • Chinese Large Model Leaderboard - an expert-driven benchmark for Chineses LLMs.
    • CompassRank - CompassRank is dedicated to exploring the most advanced language and visual models, offering a comprehensive, objective, and neutral evaluation reference for the industry and research.
    • CompMix - a benchmark evaluating QA methods that operate over a mixture of heterogeneous input sources (KB, text, tables, infoboxes).
    • DreamBench++ - a benchmark for evaluating the performance of large language models (LLMs) in various tasks related to both textual and visual imagination.
    • FELM - a meta-benchmark that evaluates how well factuality evaluators assess the outputs of large language models (LLMs).
    • InfiBench - a benchmark designed to evaluate large language models (LLMs) specifically in their ability to answer real-world coding-related questions.
    • LawBench - a benchmark designed to evaluate large language models in the legal domain.
    • LLMEval - focuses on understanding how these models perform in various scenarios and analyzing results from an interpretability perspective.
    • M3CoT - a benchmark that evaluates large language models on a variety of multimodal reasoning tasks, including language, natural and social sciences, physical and social commonsense, temporal reasoning, algebra, and geometry.
    • MathEval - a comprehensive benchmarking platform designed to evaluate large models' mathematical abilities across 20 fields and nearly 30,000 math problems.
    • MMedBench - a benchmark that evaluates large language models' ability to answer medical questions across multiple languages.
    • MMToM-QA - a multimodal question-answering benchmark designed to evaluate AI models' cognitive ability to understand human beliefs and goals.
    • OlympicArena - a benchmark for evaluating AI models across multiple academic disciplines like math, physics, chemistry, biology, and more.
    • SciBench - benchmark designed to evaluate large language models (LLMs) on solving complex, college-level scientific problems from domains like chemistry, physics, and mathematics.
    • SuperBench - a benchmark platform designed for evaluating large language models (LLMs) on a range of tasks, particularly focusing on their performance in different aspects such as natural language understanding, reasoning, and generalization.
    • SuperLim - a Swedish language understanding benchmark that evaluates natural language processing (NLP) models on various tasks such as argumentation analysis, semantic similarity, and textual entailment.
    • TAT-DQA - a large-scale Document Visual Question Answering (VQA) dataset designed for complex document understanding, particularly in financial reports.
    • TAT-QA - a large-scale question-answering benchmark focused on real-world financial data, integrating both tabular and textual information.
    • VisualWebArena - a benchmark designed to assess the performance of multimodal web agents on realistic visually grounded tasks.
    • We-Math - a benchmark that evaluates large multimodal models (LMMs) on their ability to perform human-like mathematical reasoning.
    • WHOOPS! - a benchmark dataset testing AI's ability to reason about visual commonsense through images that defy normal expectations.
    • LiveBench - A Challenging, Contamination-Free LLM Benchmark.
    • CompassRank - CompassRank is dedicated to exploring the most advanced language and visual models, offering a comprehensive, objective, and neutral evaluation reference for the industry and research.
    • LLMEval - focuses on understanding how these models perform in various scenarios and analyzing results from an interpretability perspective.
    • MathEval - a comprehensive benchmarking platform designed to evaluate large models' mathematical abilities across 20 fields and nearly 30,000 math problems.
    • Chatbot Arena Leaderboard - a benchmark platform for large language models (LLMs) that features anonymous, randomized battles in a crowdsourced manner.

  • LLM Training Frameworks

    • RLHF LLM

      • FairScale - FairScale is a PyTorch extension library for high performance and large scale training.

      • Alpa - Alpa is a system for training and serving large-scale neural networks.
      • torchtune - A Native-PyTorch Library for LLM Fine-tuning.
      • torchtitan - A native PyTorch Library for large model training.
      • Megatron-LM - Ongoing research training transformer models at scale.
      • Colossal-AI - Making large AI models cheaper, faster, and more accessible.
      • BMTrain - Efficient Training for Big Models.
      • Mesh Tensorflow - Mesh TensorFlow: Model Parallelism Made Easier.
      • maxtext - A simple, performant and scalable Jax LLM!
      • GPT-NeoX - An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library.
      • Meta Lingua - a lean, efficient, and easy-to-hack codebase to research LLMs.
      • Litgpt - 20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
      • nanotron - Minimalistic large language model 3D-parallelism training.
      • DeepSpeed - DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
      • ROLL - An Efficient and User-Friendly Scaling Library for Reinforcement Learning with Large Language Models.
      • veRL - veRL is a flexible and efficient RL framework for LLMs.
      • NeMo Framework - Generative AI framework built for researchers and PyTorch developers working on Large Language Models (LLMs), Multimodal Models (MMs), Automatic Speech Recognition (ASR), Text to Speech (TTS), and Computer Vision (CV) domains.
      • Transformer Engine - A library for accelerating Transformer model training on NVIDIA GPUs.
      • OpenRLHF - An Easy-to-use, Scalable and High-performance RLHF Framework (70B+ PPO Full Tuning & Iterative DPO & LoRA & RingAttention & RFT).
      • TRL - TRL is a full stack library where we provide a set of tools to train transformer language models with Reinforcement Learning, from the Supervised Fine-tuning step (SFT), Reward Modeling step (RM) to the Proximal Policy Optimization (PPO) step.
      • Axolotl - Open-source framework for fine-tuning and evaluating LLMs. It simplifies the process of experimenting with different training configurations and makes it easy to reproduce and share results, supporting features like LoRA, QLoRA, DeepSpeed, PEFT, and multi-GPU setups.

  • LLM Tutorials and Courses

Programming Languages
Python 79 TypeScript 12 Jupyter Notebook 9 JavaScript 6 Rust 4 C++ 4 C# 2 Go 2 CSS 1 HTML 1
Categories
Open LLM 91 Milestone Papers 78 LLM Applications 41 LLM Deployment 38 LLM Leaderboard 33 Open LLM --> 30 Other Papers 27 Courses 22 LLM Training Frameworks 21 Tutorials 20 Trending LLM Projects 19 Great thoughts about LLM 15 LLM Tutorials and Courses 15 Miscellaneous 10 Opinions 9 LLM Evaluation: 9 LLM Books 6 LLM Inference 6 LLM Data 5 Prompting libraries & tools 5 Other Useful Resources 4 Books 1 Deploying Tools 1
Sub Categories
RLHF LLM 96 Instruction finetuned LLM 7
Keywords
llm 50 large-language-models 21 ai 20 llama 18 chatgpt 16 openai 15 llms 14 generative-ai 13 llmops 12 gpt 11 deep-learning 11 machine-learning 10 prompt-engineering 10 chatbot 9 nlp 9 evaluation 8 rag 8 python 8 llm-inference 7 deepseek 7 langchain 7 qwen 6 inference 6 mistral 6 llama3 6 awesome 6 transformers 6 artificial-intelligence 5 pytorch 5 agents 5 llm-serving 5 llama2 5 agent 5 cuda 4 llm-evaluation 4 transformer 4 gpt-3 4 mlops 4 gpt-4 4 finetuning 4 language-model 4 framework 4 fine-tuning 4 gpu 4 analytics 3 embeddings 3 llama-index 3 llm-observability 3 observability 3 data 3