Projects in Awesome Lists tagged with llm-inference

https://github.com/nomic-ai/gpt4all

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

Last synced: 24 Sep 2025

https://github.com/ray-project/ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

data-science deep-learning deployment distributed hyperparameter-optimization hyperparameter-search large-language-models llm llm-inference llm-serving machine-learning optimization parallel python pytorch ray reinforcement-learning rllib serving tensorflow

Last synced: 19 Feb 2026

https://microsoft.github.io/autogen/

A programming framework for agentic AI 🤖

agent-based-framework agent-oriented-programming agentic agentic-agi chat chat-application chatbot chatgpt gpt gpt-35-turbo gpt-4 llm-agent llm-framework llm-inference llmops

Last synced: 08 May 2025

https://github.com/gitleaks/gitleaks

Find secrets with Gitleaks 🔑

ai-powered ci-cd cicd cli data-loss-prevention devsecops dlp git gitleaks go golang hacktoberfest llm llm-inference llm-training open-source secret security security-tools

Last synced: 15 Mar 2026

https://github.com/liguodongiot/llm-action

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

llm llm-inference llm-serving llm-training llmops

Last synced: 15 May 2025

https://github.com/lightning-ai/litgpt

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

ai artificial-intelligence deep-learning large-language-models llm llm-inference llms

Last synced: 12 May 2025

https://github.com/Lightning-AI/litgpt

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

ai artificial-intelligence deep-learning large-language-models llm llm-inference llms

Last synced: 26 Mar 2025

https://github.com/bentoml/openllm

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

bentoml fine-tuning llama llama2 llama3-1 llama3-2 llama3-2-vision llm llm-inference llm-ops llm-serving llmops mistral mlops model-inference open-source-llm openllm vicuna

Last synced: 23 Oct 2025

https://github.com/mistralai/mistral-inference

Official inference library for Mistral models

llm llm-inference mistralai

Last synced: 12 May 2025

https://github.com/mistralai/mistral-inference?tab=readme-ov-file

Official inference library for Mistral models

llm llm-inference mistralai

Last synced: 16 Mar 2025

https://github.com/bentoml/OpenLLM

Run any open-source LLMs, such as Llama 3.1, Gemma, as OpenAI compatible API endpoint in the cloud.

bentoml fine-tuning llama llama2 llama3-1 llama3-2 llama3-2-vision llm llm-inference llm-ops llm-serving llmops mistral mlops model-inference open-source-llm openllm vicuna

Last synced: 14 Mar 2025

https://github.com/openvinotoolkit/openvino

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

ai computer-vision deep-learning deploy-ai diffusion-models generative-ai good-first-issue inference llm-inference natural-language-processing nlp openvino optimize-ai performance-boost recommendation-system speech-recognition stable-diffusion transformers yolo

Last synced: 12 May 2025

https://github.com/sjtu-ipads/powerinfer

High-speed Large Language Model Serving for Local Deployment

large-language-models llama llm llm-inference local-inference

Last synced: 12 May 2025

https://github.com/SJTU-IPADS/PowerInfer

High-speed Large Language Model Serving on PCs with Consumer-grade GPUs

bamboo-7b falcon large-language-models llama llm llm-inference local-inference

Last synced: 18 Mar 2025

https://github.com/bentoml/bentoml

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

ai-inference deep-learning generative-ai inference-platform llm llm-inference llm-serving llmops machine-learning ml-engineering mlops model-inference-service model-serving multimodal python

Last synced: 06 Mar 2026

https://github.com/bentoml/BentoML

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and much more!

ai-inference deep-learning generative-ai inference-platform llm llm-inference llm-serving llmops machine-learning ml-engineering mlops model-inference-service model-serving multimodal python

Last synced: 12 Mar 2025

https://github.com/internlm/lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

codellama cuda-kernels deepspeed fastertransformer internlm llama llama2 llama3 llm llm-inference turbomind

Last synced: 04 Feb 2026

https://github.com/InternLM/lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

codellama cuda-kernels deepspeed fastertransformer internlm llama llama2 llama3 llm llm-inference turbomind

Last synced: 20 Mar 2025

https://github.com/kserve/kserve

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

artificial-intelligence cncf genai hacktoberfest istio k8s knative kserve kubeflow kubernetes llm-inference machine-learning mlops model-interpretability model-serving pytorch service-mesh tensorflow vllm xgboost

Last synced: 15 Jul 2026

https://github.com/flashinfer-ai/flashinfer

FlashInfer: Kernel Library for LLM Serving

attention cuda distributed-inference gpu jit large-large-models llm-inference moe nvidia pytorch

Last synced: 02 Jul 2026

https://github.com/superduper-io/superduper

Superduper: End-to-end framework for building custom AI applications and agents.

ai chatbot data database distributed-ml inference llm-inference llm-serving llmops ml mlops mongodb pretrained-models python pytorch rag semantic-search torch transformers vector-search

Last synced: 14 May 2025

https://github.com/gpustack/gpustack

A GPU cluster manager that configures and orchestrates inference engines like vLLM and SGLang for high-performance AI model deployment.

ascend cuda deepseek distributed-inference genai high-performance-inference inference llama llm llm-inference llm-serving maas mindie openai qwen rocm sglang vllm

Last synced: 20 Apr 2026

https://github.com/bojieli/ai-infra-book

《深入理解 AI Infra：量化分析与系统设计》（李博杰著）开源书稿：从硬件约束和模型架构出发，量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验

accelerator ai-infra ai-infrastructure book datacenter-network deepseek distributed-systems gpu kv-cache llm llm-inference llm-serving llm-training mixture-of-experts performance-engineering

Last synced: 26 Sep 2026

https://github.com/ruvnet/ruvector

RuVector is a High Performance, Real-Time, Self-Learning Ai, Vector GNN, Memory DB built in Rust.

ai ai-ocr attention-mechanism gnn gnn-model gnns graph graph-neural-networks llm-inference low-latency mincut neo4j ocr onnx rust vector wasm

Last synced: 27 Aug 2026

https://github.com/algorithmicsuperintelligence/optillm

Optimizing inference proxy for LLMs

agent agentic-ai agentic-framework agentic-workflow agents api-gateway chain-of-thought genai large-language-models llm llm-inference llmapi mixture-of-experts moa monte-carlo-tree-search openai openai-api optimization prompt-engineering proxy-server

Last synced: 05 Jul 2026

https://github.com/fellouai/eko

Eko (Eko Keeps Operating) - Build Production-ready Agentic Workflow with Natural Language - eko.fellou.ai

agent agentic-ai agentic-ai-development agentic-framework agentic-workflow agents ai-agents browser-automation browseruse chain-of-thought computer-automation computeruse genai llm-agents llm-inference llmapi natural-language-inference prompt-engineering rag workflow

Last synced: 29 Dec 2025

https://github.com/neuralmagic/deepsparse

Sparsity-aware deep learning inference runtime for CPUs

computer-vision cpus deepsparse inference llm-inference machinelearning nlp object-detection onnx performance pretrained-models pruning quantization sparsification

Last synced: 14 May 2025

https://github.com/nvidia/generativeaiexamples

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

gpu-acceleration large-language-models llm llm-inference microservice nemo rag retrieval-augmented-generation tensorrt triton-inference-server

Last synced: 13 May 2025

https://github.com/predibase/lorax

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

fine-tuning gpt llama llm llm-inference llm-serving llmops lora model-serving pytorch transformers

Last synced: 12 May 2025

https://github.com/NVIDIA/GenerativeAIExamples

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

gpu-acceleration large-language-models llm llm-inference microservice nemo rag retrieval-augmented-generation tensorrt triton-inference-server

Last synced: 28 Mar 2025

https://github.com/databricks/dbrx

Code examples and resources for DBRX, a large language model developed by Databricks

databricks gen-ai generative-ai llm llm-inference llm-training mosaic-ai

Last synced: 25 Oct 2025

https://github.com/fasterdecoding/medusa

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

llm llm-inference

Last synced: 14 May 2025

https://github.com/FasterDecoding/Medusa

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

llm llm-inference

Last synced: 09 May 2025

https://github.com/codelion/openevolve

Open-source implementation of AlphaEvolve

alpha-evolve alphacode alphaevolve coding-agent deepmind deepmind-lab discovery distributed-evolutionary-algorithms evolutionary-algorithms evolutionary-computation genetic-algorithm genetic-algorithms iterative-methods iterative-refinement llm-engineering llm-ensemble llm-inference openevolve optimize

Last synced: 10 Jun 2025

https://github.com/intel/intel-extension-for-transformers

⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡

4-bits autoround chatbot chatpdf gaudi3 habana intel-optimized-llamacpp large-language-model llm-cpu llm-inference neural-chat neural-chat-7b rag retrieval speculative-decoding streamingllm

Last synced: 03 Jul 2026

https://github.com/lemonade-sdk/lemonade

Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk

ai amd genai gpu llama llm llm-inference local-server mcp mcp-server mistral npu onnxruntime openai-api qwen radeon rocm ryzen vulkan

Last synced: 02 Apr 2026

https://github.com/microsoft/aici

AICI: Prompts as (Wasm) Programs

ai inference language-model llm llm-framework llm-inference llm-serving llmops model-serving rust transformer wasm wasmtime

Last synced: 14 May 2025

https://github.com/b4rtaz/distributed-llama

Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.

distributed-computing distributed-llm llama2 llama3 llm llm-inference llms neural-network open-llm

Last synced: 13 Apr 2025

https://github.com/liltom-eth/llama2-webui

Run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac). Use `llama2-wrapper` as your local llama2 backend for Generative Agents/Apps.

llama-2 llama2 llm llm-inference

Last synced: 14 May 2025

https://github.com/neuron-core/neuron-ai

The PHP Agentic Framework to build production-ready AI driven applications. Connect components (LLMs, vector DBs, memory) to agents that can interact with your data. With its modular architecture it's best suited for building RAG, multi-agent workflows, or business process automations.

agent agentic-ai agentic-framework agents ai llm llm-inference llms php vector-database

Last synced: 01 May 2026

https://github.com/beam-cloud/beta9

Ultrafast serverless GPU inference, sandboxes, and background jobs

autoscaler cloudrun cuda developer-productivity distributed-computing faas fine-tuning functions-as-a-service generative-ai gpu large-language-models llm llm-inference ml-platform paas self-hosted serverless serverless-containers

Last synced: 19 Jul 2026

https://github.com/xLLM-AI/xllm

A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.

deepseek glm inference inference-engine large-language-models llm-inference qwen

Last synced: 05 Aug 2026

https://github.com/datawhalechina/zero-to-sglang

Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.

cuda datawhale inference-engine infra kv-cache llm llm-inference sglang tutorial

Last synced: 26 Sep 2026

https://github.com/ray-project/aviary

RayLLM - LLMs on Ray

distributed-systems large-language-models llm llm-inference llm-serving llmops ray serving transformers

Last synced: 12 Jan 2026

https://github.com/ray-project/ray-llm

RayLLM - LLMs on Ray

distributed-systems large-language-models llm llm-inference llm-serving llmops ray serving transformers

Last synced: 19 Jul 2026

https://github.com/character-ai/prompt-poet

Streamlines and simplifies prompt design for both developers and non-technical users with a low code approach.

llm llm-inference prompt prompt-design prompt-engineering prompt-tuning prompting

Last synced: 14 Jan 2026

https://github.com/sauravpanda/browserai

Run local LLMs like llama, deepseek-distill, kokoro and more inside your browser

agents ai llama llm llm-inference local localllm tts webgpu

Last synced: 18 Apr 2026

https://github.com/PySpur-Dev/PySpur

Graph-Based Editor for LLM Workflows

agent agents ai javascript llm llm-inference openai python react workflow

Last synced: 14 Sep 2025

https://github.com/cloud-code-ai/browserai

Run local LLMs like llama, deepseek-distill, kokoro and more inside your browser

agents ai llama llm llm-inference local localllm tts webgpu

Last synced: 14 Apr 2025

https://github.com/PySpur-Dev/pyspur

Graph-Based Editor for LLM Workflows

agent agents ai javascript llm llm-inference openai python react workflow

Last synced: 07 Sep 2025

https://github.com/lean-dojo/LeanCopilot

LLMs as Copilots for Theorem Proving in Lean

formal-mathematics lean lean4 llm-inference machine-learning theorem-proving

Last synced: 09 Jul 2025

https://github.com/dalisoft/awesome-hosting?tab=readme-ov-file

List of awesome hosting sorted by minimal plan price

ai clawdbot cloud database deepseek-r1 free hosting iaas linux llm llm-inference moltbot nodejs openclaw saas serverless vps windows

Last synced: 15 Sep 2026

https://github.com/tingaicompass/AI-Compass

“AI-Compass”将为社区指引在 AI 技术海洋中航行的方向，无论你是初学者还是进阶开发者，都能在这里找到通往 AI 各大方向的路径。旨在帮助开发者系统性地了解 AI 的核心概念、主流技术、前沿趋势，并通过实践掌握从理论到落地的全过程。

agent ai llm llm-inference llm-training nlp rl rlhf

Last synced: 02 Sep 2026

https://github.com/zhihu/zhilight

A highly optimized LLM inference acceleration engine for Llama and its variants.

cuda deepseek-r1 gpt inference-engine llama llm llm-inference llm-serving model-serving pytorch

Last synced: 15 May 2025

https://github.com/harleyszhang/llm_note

LLM notes, including model inference, transformer model structure, and llm framework code analysis notes.

cuda-programming kv-cache llm llm-inference transformer-models triton-kernels vllm

Last synced: 23 Aug 2025

https://github.com/foldl/chatllm.cpp

Pure C++ implementation of several models for real-time chatting on your computer (CPU & GPU)

llm llm-inference

Last synced: 10 Feb 2026

https://github.com/SafeAILab/EAGLE

Official Implementation of EAGLE-1 (ICML'24) and EAGLE-2 (EMNLP'24)

large-language-models llm-inference speculative-decoding

Last synced: 20 Mar 2025

https://github.com/katanemo/archgw

Arch is an intelligent prompt gateway. Engineered with (fast) LLMs for the secure handling, robust observability, and seamless integration of prompts with your APIs - outside business logic. Built by the core contributors of Envoy proxy, on Envoy.

ai-gateway envoy envoyproxy gateway generative-ai llm-gateway llm-inference llm-routing llmops llms openai prompt proxy proxy-server routing

Last synced: 21 Oct 2025

https://github.com/inspector-apm/neuron-ai

The PHP Agent Development Kit - powered by Inspector.dev

agent agentic-ai agentic-framework agents ai llm llm-inference llms php vector-database

Last synced: 21 Jun 2025

https://github.com/stoyan-stoyanov/llmflows

LLMFlows - Simple, Explicit and Transparent LLM Apps

ai chatgpt gpt-4 llm llm-inference llmops llms machine-learning openai prompt-engineering python question-answering vector-database

Last synced: 14 May 2025

https://github.com/mukel/llama3.java

Practical Llama 3 inference in Java

chatgpt genai gguf huggingface java llama llama3 llamacpp llm llm-inference llms openai simd transformers

Last synced: 15 May 2025

https://github.com/pegainfer-project/pegainfer

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

cuda cuda-kernels deepseek gpu inference inference-engine kimi kimi-k2 kv-cache llm llm-inference llm-serving model-serving moe openai-api paged-attention qwen qwen3 rust vllm

Last synced: 14 Sep 2026

https://github.com/eastriverlee/llm.swift

LLM.swift is a simple and readable library that allows you to interact with large language models locally with ease for macOS, iOS, watchOS, tvOS, and visionOS.

gguf ios llm llm-inference macos swift tvos visionos watchos

Last synced: 15 May 2025

https://github.com/run-ai/genv

GPU environment and cluster management with LLM support

bash container-runtime containers data-science deep-learning docker gpu gpus jupyter-notebook jupyterlab-extension k8s kubernetes llm-inference llms nvidia-gpu ollama ray vscode vscode-extension zsh

Last synced: 16 May 2025

https://github.com/rohan-paul/LLM-FineTuning-Large-Language-Models

LLM (Large Language Model) FineTuning

gpt-3 gpt3-turbo large-language-models llama2 llm llm-finetuning llm-inference llm-serving llm-training mistral-7b open-source-llm pytorch

Last synced: 16 Oct 2025

https://github.com/zeux/calm

CUDA/Metal accelerated language model inference

cuda llm-inference ml

Last synced: 10 Apr 2025

https://github.com/timmyy123/LLM-Hub

Local AI Assistant

ai gemma gemma4 gemma4-agent-skills gptoss granite lfm25 llama llm llm-inference mistral phi4 rag stable-diffusion whisper

Last synced: 12 Aug 2026

https://github.com/eastriverlee/LLM.swift

LLM.swift is a simple and readable library that allows you to interact with large language models locally with ease for macOS, iOS, watchOS, tvOS, and visionOS.

gguf ios llm llm-inference macos swift tvos visionos watchos

Last synced: 11 Mar 2025

https://github.com/rohan-paul/llm-finetuning-large-language-models

LLM (Large Language Model) FineTuning

gpt-3 gpt3-turbo large-language-models llama2 llm llm-finetuning llm-inference llm-serving llm-training mistral-7b open-source-llm pytorch

Last synced: 04 Apr 2025

https://github.com/nano-collective/nanocoder

A beautiful local-first coding agent running in your terminal - built by the community for the community ⚒

ai ai-agents ai-coding coding-agents llm llm-inference ollama openai openrouter

Last synced: 10 Feb 2026

https://github.com/stanford-mast/blast

Browser-LLM Auto-Scaling Technology

ai-agents browser-automation llm-inference python

Last synced: 11 May 2025

https://github.com/TilmanGriesel/chipper

✨ AI interface for tinkerers (Ollama, Haystack RAG, Python)

agent agentic-ai deepseek deepseek-chat deepseek-r1 embedding hugging-face huggingface llama3 llm llm-inference ollama ollama-api ollama-client ollama-gui phi4 rag retrival-augmented-generation

Last synced: 29 Aug 2026

https://github.com/michael-a-kuykendall/shimmy

⚡ Python-free Rust inference server — OpenAI-API compatible. GGUF + SafeTensors, hot model swap, auto-discovery, single binary. FREE now, FREE forever.

api-server command-line-tool developer-tools gguf huggingface huggingface-models huggingface-transformers inference-server llama llamacpp llm-inference local-ai lora machine-learning ollama-api openai-compatible rust rust-crate transformers

Last synced: 13 Sep 2025

https://github.com/feifeibear/long-context-attention

USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference

attention-is-all-you-need deepspeed-ulysses llm-inference llm-training pytorch ring-attention

Last synced: 14 May 2025

https://github.com/Nano-Collective/nanocoder

A beautiful local-first coding agent running in your terminal - built by the community for the community ⚒

ai ai-agents ai-coding coding-agents llm llm-inference ollama openai openrouter

Last synced: 23 Sep 2025

https://github.com/hpcaitech/swiftinfer

Efficient AI Inference & Serving

artificial-intelligence deep-learning gpt inference llama llama2 llm-inference llm-serving

Last synced: 05 Apr 2025

https://github.com/ome-projects/ome

Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton

deepseek k8s kimi-k2 llama llm llm-inference model-as-a-service model-serving multi-node-kubernetes oracle-cloud pd-disaggregation qwen sglang vllm

Last synced: 09 Jun 2026

https://github.com/brontoguana/krasis

Krasis is a Hybrid LLM runtime which focuses on efficient running of larger models on consumer grade VRAM limited hardware

cpu-inference gguf-model-support gpu-inference high-performance-inference hybrid-inference inference-engine inference-optimization large-language-models llama-cpp-alternative llm-inference mixture-of-experts transformer

Last synced: 30 May 2026

https://github.com/npc-worldwide/incognide

Explore the unknown, build the future, own your data.

agents ai ai-agents artificial-intelligence ide llm-inference llms node npc npm research

Last synced: 03 Jul 2026

https://github.com/tilmangriesel/chipper

✨ AI interface for tinkerers (Ollama, Haystack RAG, Python)

agent agentic-ai deepseek deepseek-chat deepseek-r1 embedding hugging-face huggingface llama3 llm llm-inference ollama ollama-api ollama-client ollama-gui phi4 rag retrival-augmented-generation

Last synced: 16 May 2025

https://github.com/flagai-open/aquila2

The official repo of Aquila2 series proposed by BAAI, including pretrained & chat large language models.

llm llm-inference llm-training

Last synced: 15 May 2025

https://github.com/kenza-ai/sagify

LLMs and Machine Learning done easily

ai-gateway anthropic cohere generative-ai langchain langchain-python large-language-model large-language-models llm llm-inference llmops open-source-llm openai sagemaker

Last synced: 16 May 2025

https://github.com/vectorch-ai/ScaleLLM

A high-performance inference system for large language models, designed for production environments.

cuda efficiency gpu inference llama llama3 llm llm-inference model performance production serving speculative transformer

Last synced: 09 May 2025

https://github.com/vectorch-ai/scalellm

A high-performance inference system for large language models, designed for production environments.

cuda efficiency gpu inference llama llama3 llm llm-inference model performance production serving speculative transformer

Last synced: 14 Apr 2025

https://github.com/rizerphe/local-llm-function-calling

A tool for generating function arguments and choosing what function to call with local LLMs

chatgpt-functions huggingface-transformers json-schema llm llm-inference openai-function-call openai-functions

Last synced: 26 Oct 2025

https://github.com/FlagAI-Open/Aquila2

The official repo of Aquila2 series proposed by BAAI, including pretrained & chat large language models.

llm llm-inference llm-training

Last synced: 08 Apr 2025

https://github.com/huawei-csl/kvarn

KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.

agentic-ai kv-cache llm llm-inference long-context quantization vllm

Last synced: 01 Jul 2026

https://github.com/preternatural-explore/mlx-swift-chat

A multi-platform SwiftUI frontend for running local LLMs with Apple's MLX framework.

ios llm-inference macos mlx mlx-swift swiftui

Last synced: 10 Apr 2025

https://github.com/sgl-project/ome

Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton

deepseek k8s kimi-k2 llama llm llm-inference model-as-a-service model-serving multi-node-kubernetes oracle-cloud pd-disaggregation qwen sglang vllm

Last synced: 17 Mar 2026

https://github.com/jax-ml/scaling-book

Home for "How To Scale Your Model", a short blog-style textbook about scaling LLMs on TPUs

jax llm-inference llms roofline tpus

Last synced: 18 Jun 2025

https://github.com/felladrin/minisearch

Minimalist web-searching platform with an AI assistant that runs directly from your browser. Uses WebLLM, Wllama and SearXNG. Demo: https://felladrin-minisearch.hf.space

ai ai-search-engine artificial-intelligence generative-ai gpu-accelerated information-retrieval llm llm-inference metasearch metasearch-engine perplexity perplexity-ai question-answering rag retrieval-augmented-generation searxng web-llm web-search webapp wllama

Last synced: 12 Apr 2025

https://github.com/EulerSearch/embedding_studio

Embedding Studio is a framework which allows you transform your Vector Database into a feature-rich Search Engine.

embeddings embeddings-similarity fine-tuning llm-inference query-parser search-algorithm search-engine search-query-parser semantic-similarity unstructured-data unstructured-search vector-database

Last synced: 06 Aug 2025

https://github.com/nvidia/star-attention

Efficient LLM Inference over Long Sequences

attention-mechanism large-language-models llm-inference

Last synced: 16 May 2025

https://github.com/felladrin/MiniSearch

Minimalist web-searching platform with an AI assistant that runs directly from your browser. Uses WebLLM, Wllama and SearXNG. Demo: https://felladrin-minisearch.hf.space

ai ai-search-engine artificial-intelligence generative-ai gpu-accelerated information-retrieval llm llm-inference metasearch metasearch-engine perplexity perplexity-ai question-answering rag retrieval-augmented-generation searxng web-llm web-search webapp wllama