Projects in Awesome Lists tagged with dynamic-batching
A curated list of projects in awesome lists tagged with dynamic-batching .
https://github.com/netease-media/grps
Deep Learning Deployment Framework: Supports tf/torch/trt/trtllm/vllm and other NN frameworks. Support dynamic batching, and streaming modes. It is dual-language compatible with Python and C++, offering scalability, extensibility, and high performance. It helps users quickly deploy models and provide services through HTTP/RPC interfaces.
dynamic-batching serving tensorflow tensorrt tensorrt-llm torch triton-inference-server vllm
Last synced: 05 Apr 2025
https://github.com/NetEase-Media/grps
【深度学习模型部署框架】支持tf/torch/trt/trtllm/vllm以及更多nn框架，支持dynamic batching、streaming模式，支持python/c++双语言，可限制，可拓展，高性能。帮助用户快速地将模型部署到线上，并通过http/rpc接口方式提供服务。
dynamic-batching serving tensorflow tensorrt tensorrt-llm torch triton-inference-server vllm
Last synced: 04 Nov 2025
https://github.com/microsoft/batch-inference
Dynamic batching library for Deep Learning inference. Tutorials for LLM, GPT scenarios.
deep-learning dynamic-batching gpt inference llm performance-optimization python
Last synced: 30 Apr 2025
https://github.com/chncwang/insnet
InsNet Runs Instance-dependent Neural Networks with Padding-free Dynamic Batching.
deep-learning-library dynamic-batching nlp
Last synced: 14 Jan 2026
https://github.com/chncwang/InsNet
InsNet Runs Instance-dependent Neural Networks with Padding-free Dynamic Batching.
deep-learning-library dynamic-batching nlp
Last synced: 17 Nov 2025
https://github.com/kimmmmyy223/llm-batch
🚀 Process JSON data in batches with `llm-batch`, leveraging sequential or parallel modes for efficient interaction with LLMs.
aws batch-inference bedrock deep-learning distributed-computing dynamic-batching flask language-model large-language-models llm-agent llm-inference nlp ops python rabbitmq ray-data react vllm
Last synced: 08 Apr 2026
https://github.com/theibrahimstudio/finder.livebatch
A lightweight, framework-agnostic middleware that dynamically batches inference requests in real time to maximize GPU/TPU utilization.
dynamic-batching golang gpu grpc microservices ml-inference model-serving performance-optimization
Last synced: 18 May 2026
https://github.com/bendangnuksung/dynabatch
dynabatch is a PyTorch/Hugging Face batching utility that sorts variable-length text by difficulty, then dynamically increases batch size on easier samples using a pre-trained VRAM predictor to improve GPU utilization and throughput while reducing OOM risk with fallback handling.
dataloader dynamic-batching huggingface huggingface-trainer machine-translation pytorch sampler transformers vram-optimization
Last synced: 30 May 2026
