Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

awesome-cuda-triton-hpc

🔥🔥🔥 A collection of some awesome public CUDA, cuBLAS, cuDNN, CUTLASS, TensorRT, TensorRT-LLM, Triton, TVM, MLIR and High Performance Computing (HPC) projects.
https://github.com/coderonion/awesome-cuda-triton-hpc

Last synced: 6 days ago
JSON representation

Blogs
Frameworks
- yetone/openai-translator - translator?style=social"/> : The translator that does more than just translation - powered by OpenAI.
- go-skynet/LocalAI - skynet/LocalAI?style=social"/> : 🤖 Self-hosted, community-driven, local OpenAI-compatible API. Drop-in replacement for OpenAI running LLMs on consumer-grade hardware. Free Open Source OpenAI alternative. No GPU required. LocalAI is an API to run ggml compatible models: llama, gpt4all, rwkv, whisper, vicuna, koala, gpt4all-j, cerebras, falcon, dolly, starcoder, and many other. [localai.io](https://localai.io/)
- wangzhaode/ChatGLM-MNN - MNN?style=social"/> : Pure C++, Easy Deploy ChatGLM-6B.
- MUTLASS
- MooreThreads/torch_musa
- FlagGems - performance general operator library implemented in [OpenAI Triton](https://github.com/openai/triton). It aims to provide a suite of kernel functions to accelerate LLM training and inference.
- FlagPerf - source software platform for benchmarking AI chips. FlagPerf是智源研究院联合AI硬件厂商共建的一体化AI硬件评测引擎，旨在建立以产业实践为导向的指标体系，评测AI硬件在软件栈组合（模型+框架+编译器）下的实际能力。
- SCUDA - over-IP. SCUDA is a GPU over IP bridge allowing GPUs on remote machines to be attached to CPU-only machines.
- Burn - rs/burn?style=social"/> : Burn - A Flexible and Comprehensive Deep Learning Framework in Rust. [burn-rs.github.io/](https://burn-rs.github.io/)
- BLAS - vector operations, the Level 2 BLAS perform matrix-vector operations, and the Level 3 BLAS perform matrix-matrix operations.
- BLAS - vector operations, the Level 2 BLAS perform matrix-vector operations, and the Level 3 BLAS perform matrix-matrix operations.
- LAPACK - LAPACK/lapack?style=social"/> : LAPACK development repository. [LAPACK](https://www.netlib.org/lapack/) — Linear Algebra PACKage. LAPACK is written in Fortran 90 and provides routines for solving systems of simultaneous linear equations, least-squares solutions of linear systems of equations, eigenvalue problems, and singular value problems. The associated matrix factorizations (LU, Cholesky, QR, SVD, Schur, generalized Schur) are also provided, as are related computations such as reordering of the Schur factorizations and estimating condition numbers. Dense and banded matrices are handled, but not general sparse matrices. In all areas, similar functionality is provided for real and complex matrices, in both single and double precision.
- OpenBLAS
- BLIS - like Library Instantiation Software Framework.
- NumPy
- SciPy - source software for mathematics, science, and engineering. It includes modules for statistics, optimization, integration, linear algebra, Fourier transforms, signal and image processing, ODE solvers, and more. [scipy.org](https://scipy.org/)
- Gonum
- CCCL - quality, high-performance, and easy-to-use C++ abstractions for CUDA developers.
- HIP - Compute Interface for Portability. HIP is a C++ Runtime API and Kernel Language that allows developers to create portable applications for AMD and NVIDIA GPUs from single source code. [rocmdocs.amd.com/projects/HIP/](https://rocmdocs.amd.com/projects/HIP/)
- PyCUDA
- jessfraz/advent-of-cuda - of-cuda?style=social"/> : Doing advent of code with CUDA and rust.
- Bend - level programming language.[higherorderco.com](https://higherorderco.com/)
- HVM
- ZLUDA
- Rust-CUDA - GPU/Rust-CUDA?style=social"/> : Ecosystem of libraries and tools for writing and executing fast GPU code fully in Rust.
- cudarc
- bindgen_cuda - lang/rust-bindgen) in philosophy. It will help create automatic bindgen to cuda kernels source files and make them easier to use directly from Rust.
- cuda-driver - driver?style=social"/> : 基于 CUDA Driver API 的 cuda 运行时环境。
- async-cuda - ai/async-cuda?style=social"/> : Asynchronous CUDA for Rust.
- async-tensorrt - ai/async-tensorrt?style=social"/> : Asynchronous TensorRT for Rust.
- krnl - r-earp/krnl?style=social"/> : Safe, portable, high performance compute (GPGPU) kernels.
- custos
- spinorml/nvlib
- DoeringChristian/cuda-rs - rs?style=social"/> : Cuda Bindings for rust generated with bindgen-cli (similar to cust_raw).
- romankoblov/rust-nvrtc - nvrtc?style=social"/> : NVRTC bindings for RUST.
- solkitten/astro-cuda - cuda?style=social"/> : CUDA Driver API bindings for Rust.
- BLIS - like Library Instantiation Software Framework.
- LAPACK - LAPACK/lapack?style=social"/> : LAPACK development repository. [LAPACK](https://www.netlib.org/lapack/) — Linear Algebra PACKage. LAPACK is written in Fortran 90 and provides routines for solving systems of simultaneous linear equations, least-squares solutions of linear systems of equations, eigenvalue problems, and singular value problems. The associated matrix factorizations (LU, Cholesky, QR, SVD, Schur, generalized Schur) are also provided, as are related computations such as reordering of the Schur factorizations and estimating condition numbers. Dense and banded matrices are handled, but not general sparse matrices. In all areas, similar functionality is provided for real and complex matrices, in both single and double precision.
- OpenBLAS
- NumPy
- SciPy - source software for mathematics, science, and engineering. It includes modules for statistics, optimization, integration, linear algebra, Fourier transforms, signal and image processing, ODE solvers, and more. [scipy.org](https://scipy.org/)
- Gonum
- CCCL - quality, high-performance, and easy-to-use C++ abstractions for CUDA developers.
- HIP - Compute Interface for Portability. HIP is a C++ Runtime API and Kernel Language that allows developers to create portable applications for AMD and NVIDIA GPUs from single source code. [rocmdocs.amd.com/projects/HIP/](https://rocmdocs.amd.com/projects/HIP/)
- PyCUDA
- jessfraz/advent-of-cuda - of-cuda?style=social"/> : Doing advent of code with CUDA and rust.
- Bend - level programming language.[higherorderco.com](https://higherorderco.com/)
- HVM
- ZLUDA
- cudarc
- bindgen_cuda - lang/rust-bindgen) in philosophy. It will help create automatic bindgen to cuda kernels source files and make them easier to use directly from Rust.
- cuda-driver - driver?style=social"/> : 基于 CUDA Driver API 的 cuda 运行时环境。
- async-cuda - ai/async-cuda?style=social"/> : Asynchronous CUDA for Rust.
- async-tensorrt - ai/async-tensorrt?style=social"/> : Asynchronous TensorRT for Rust.
- krnl - r-earp/krnl?style=social"/> : Safe, portable, high performance compute (GPGPU) kernels.
- custos
- spinorml/nvlib
- DoeringChristian/cuda-rs - rs?style=social"/> : Cuda Bindings for rust generated with bindgen-cli (similar to cust_raw).
- romankoblov/rust-nvrtc - nvrtc?style=social"/> : NVRTC bindings for RUST.
- solkitten/astro-cuda - cuda?style=social"/> : CUDA Driver API bindings for Rust.
- bokutotu/curs
- rust-cuda/cuda-sys - cuda/cuda-sys?style=social"/> : Rust binding to CUDA APIs.
- bheisler/RustaCUDA
- tmrob2/cuda2rust_sandpit
- PhDP/rust-cuda-template - cuda-template?style=social"/> : Simple template for Rust + CUDA.
- neka-nat/cuimage - nat/cuimage?style=social"/> : Rust implementation of image processing library with CUDA.
- yanghaku/cuda-driver-sys - driver-sys?style=social"/> : Rust binding to CUDA Driver APIs.
- Canyon-ml/canyon-sys - ml/canyon-sys?style=social"/> : Rust Bindings for Cuda, CuDNN.
- cea-hpc/HARP - hpc/HARP?style=social"/> : Small tool for profiling the performance of hardware-accelerated Rust code using OpenCL and CUDA.
- Conqueror712/CUDA-Simulator - Simulator?style=social"/> : A self-developed version of the user-mode CUDA emulator project and a learning repository for Rust.
- cszach/rust-cuda-template - cuda-template?style=social"/> : A Rust CUDA template with detailed instructions.
- exor2008/fluid-simulator - simulator?style=social"/> : Rust CUDA fluid simulator.
- chichieinstein/rustycuda
- Jafagervik/cruda - Writing rust with cuda.
- lennyerik/cutransform
- cjordan/hip-sys - sys?style=social"/> : Rust bindings for HIP.
- rust-gpu - gpu?style=social"/> : 🐉 Making Rust a first-class language and ecosystem for GPU shaders 🚧 [shader.rs](https://shader.rs/)
- wgpu - rs/wgpu?style=social"/> : Safe and portable GPU abstraction in Rust, implementing WebGPU API. [wgpu.rs](https://wgpu.rs/)
- Vulkano - rs/vulkano?style=social"/> : Safe and rich Rust wrapper around the Vulkan API. Vulkano is a Rust wrapper around [the Vulkan graphics API](https://www.vulkan.org/). It follows the Rust philosophy, which is that as long as you don't use unsafe code you shouldn't be able to trigger any undefined behavior. In the case of Vulkan, this means that non-unsafe code should always conform to valid API usage.
- Ash - rs/ash?style=social"/> : Vulkan bindings for Rust.
- ocl
- opencl3
- CUDA.jl
- AMDGPU.jl
- te42kyfo/gpu-benches - benches?style=social"/> : collection of benchmarks to measure basic GPU capabilities.
- wgpu - rs/wgpu?style=social"/> : Safe and portable GPU abstraction in Rust, implementing WebGPU API. [wgpu.rs](https://wgpu.rs/)
- Vulkano - rs/vulkano?style=social"/> : Safe and rich Rust wrapper around the Vulkan API. Vulkano is a Rust wrapper around [the Vulkan graphics API](https://www.vulkan.org/). It follows the Rust philosophy, which is that as long as you don't use unsafe code you shouldn't be able to trigger any undefined behavior. In the case of Vulkan, this means that non-unsafe code should always conform to valid API usage.
- bokutotu/curs
- rust-cuda/cuda-sys - cuda/cuda-sys?style=social"/> : Rust binding to CUDA APIs.
- bheisler/RustaCUDA
- tmrob2/cuda2rust_sandpit
- PhDP/rust-cuda-template - cuda-template?style=social"/> : Simple template for Rust + CUDA.
- neka-nat/cuimage - nat/cuimage?style=social"/> : Rust implementation of image processing library with CUDA.
- yanghaku/cuda-driver-sys - driver-sys?style=social"/> : Rust binding to CUDA Driver APIs.
- Canyon-ml/canyon-sys - ml/canyon-sys?style=social"/> : Rust Bindings for Cuda, CuDNN.
- cea-hpc/HARP - hpc/HARP?style=social"/> : Small tool for profiling the performance of hardware-accelerated Rust code using OpenCL and CUDA.
- Conqueror712/CUDA-Simulator - Simulator?style=social"/> : A self-developed version of the user-mode CUDA emulator project and a learning repository for Rust.
- cszach/rust-cuda-template - cuda-template?style=social"/> : A Rust CUDA template with detailed instructions.
- exor2008/fluid-simulator - simulator?style=social"/> : Rust CUDA fluid simulator.
- chichieinstein/rustycuda
- Jafagervik/cruda - Writing rust with cuda.
- lennyerik/cutransform
- cjordan/hip-sys - sys?style=social"/> : Rust bindings for HIP.
- rust-gpu - gpu?style=social"/> : 🐉 Making Rust a first-class language and ecosystem for GPU shaders 🚧 [shader.rs](https://shader.rs/)
- Ash - rs/ash?style=social"/> : Vulkan bindings for Rust.
- ocl
- opencl3
- CUDA.jl
- AMDGPU.jl
- te42kyfo/gpu-benches - benches?style=social"/> : collection of benchmarks to measure basic GPU capabilities.
- PyTorch
- PaddlePaddle
- CUTLASS
- MatX - GPU-Accelerated Numerical Computing in Modern C++. An efficient C++17 GPU numerical computing library with Python-like syntax. [nvidia.github.io/MatX](https://nvidia.github.io/MatX)
- GenericLinearAlgebra.jl
- flashlight/flashlight
- NVlabs/tiny-cuda-nn - cuda-nn?style=social"/> : Lightning fast C++/CUDA neural network framework.
- yhwang-hub/dl_model_infer - hub/dl_model_infer?style=social"/> : his is a c++ version of the AI reasoning library. Currently, it only supports the reasoning of the tensorrt model. The follow-up plan supports the c++ reasoning of frameworks such as Openvino, NCNN, and MNN. There are two versions for pre- and post-processing, c++ version and cuda version. It is recommended to use the cuda version., This repository provides accelerated deployment cases of deep learning CV popular models, and cuda c supports dynamic-batch image process, infer, decode, NMS.
- llm.c - 2 (CPU, fp32) is ~1,000 lines of clean code in a single file. It compiles and runs instantly, and exactly matches the PyTorch reference implementation.
- llama2.c - line C file (run.c).
- gemma.cpp
- llama.cpp
- MatX - GPU-Accelerated Numerical Computing in Modern C++. An efficient C++17 GPU numerical computing library with Python-like syntax. [nvidia.github.io/MatX](https://nvidia.github.io/MatX)
- GenericLinearAlgebra.jl
- custos-math - math?style=social"/> : This crate provides CUDA, OpenCL, CPU (and Stack) based matrix operations using [custos](https://github.com/elftausend/custos).
- PyTorch
- PaddlePaddle
- flashlight/flashlight
- NVlabs/tiny-cuda-nn - cuda-nn?style=social"/> : Lightning fast C++/CUDA neural network framework.
- yhwang-hub/dl_model_infer - hub/dl_model_infer?style=social"/> : his is a c++ version of the AI reasoning library. Currently, it only supports the reasoning of the tensorrt model. The follow-up plan supports the c++ reasoning of frameworks such as Openvino, NCNN, and MNN. There are two versions for pre- and post-processing, c++ version and cuda version. It is recommended to use the cuda version., This repository provides accelerated deployment cases of deep learning CV popular models, and cuda c supports dynamic-batch image process, infer, decode, NMS.
- llm.c - 2 (CPU, fp32) is ~1,000 lines of clean code in a single file. It compiles and runs instantly, and exactly matches the PyTorch reference implementation.
- llama2.c - line C file (run.c).
- gemma.cpp
- llama.cpp
- whisper.cpp - performance inference of [OpenAI's Whisper](https://github.com/openai/whisper) automatic speech recognition (ASR) model.
- ChatGLM.cpp - plus/chatglm.cpp?style=social"/> : C++ implementation of [ChatGLM-6B](https://github.com/THUDM/ChatGLM-6B) and [ChatGLM2-6B](https://github.com/THUDM/ChatGLM2-6B).
- MegEngine/InferLLM
- whisper.cpp - performance inference of [OpenAI's Whisper](https://github.com/openai/whisper) automatic speech recognition (ASR) model.
- ChatGLM.cpp - plus/chatglm.cpp?style=social"/> : C++ implementation of [ChatGLM-6B](https://github.com/THUDM/ChatGLM-6B) and [ChatGLM2-6B](https://github.com/THUDM/ChatGLM2-6B).
- MegEngine/InferLLM
- skeskinen/llama-lite - lite?style=social"/> : Embeddings focused small version of Llama NLP model.
- Const-me/Whisper - me/Whisper?style=social"/> : High-performance GPGPU inference of OpenAI's Whisper automatic speech recognition (ASR) model.
- ztxz16/fastllm - 6B，MOSS; 可以在安卓设备上流畅运行ChatGLM-6B。
- davidar/eigenGPT
- zjhellofss/KuiperInfer (自制深度学习推理框架) - performance deep learning inference library step by step.
- ztxz16/fastllm - 6B，MOSS; 可以在安卓设备上流畅运行ChatGLM-6B。
- davidar/eigenGPT
- skeskinen/llama-lite - lite?style=social"/> : Embeddings focused small version of Llama NLP model.
- Const-me/Whisper - me/Whisper?style=social"/> : High-performance GPGPU inference of OpenAI's Whisper automatic speech recognition (ASR) model.
- Tlntin/Qwen-TensorRT-LLM - TensorRT-LLM?style=social"/> : 使用TRT-LLM完成对Qwen-7B-Chat实现推理加速。
- FeiGeChuanShu/trt2023 - 7B用TensorRT-LLM模型搭建及优化。
- TRT2022/trtllm-llama - llama?style=social"/> : ☢️ TensorRT 2023复赛——基于TensorRT-LLM的Llama模型推断加速优化。
- llama2.mojo
- dorjeduck/llm.mojo
- Candle
- Safetensors
- tch-rs - rs?style=social"/> : Rust bindings for the C++ api of PyTorch.
- Tlntin/Qwen-TensorRT-LLM - TensorRT-LLM?style=social"/> : 使用TRT-LLM完成对Qwen-7B-Chat实现推理加速。
- FeiGeChuanShu/trt2023 - 7B用TensorRT-LLM模型搭建及优化。
- TRT2022/trtllm-llama - llama?style=social"/> : ☢️ TensorRT 2023复赛——基于TensorRT-LLM的Llama模型推断加速优化。
- llama2.mojo
- dorjeduck/llm.mojo
- Candle
- Safetensors
- Tokenizers - of-the-Art Tokenizers optimized for Research and Production. [huggingface.co/docs/tokenizers](https://huggingface.co/docs/tokenizers/index)
- dfdx
- luminal
- crabml
- rustai-solutions/candle_demo_openchat_35 - solutions/candle_demo_openchat_35?style=social"/> : candle_demo_openchat_35.
- llama2.rs
- Llama2-burn - burn?style=social"/> : Llama2 LLM ported to Rust burn.
- gaxler/llama2.rs
- whisper-burn - burn?style=social"/> : A Rust implementation of OpenAI's Whisper model using the burn framework.
- stable-diffusion-burn - diffusion-burn?style=social"/> : Stable Diffusion v1.4 ported to Rust's burn framework.
- coreylowman/llama-dfdx - dfdx?style=social"/> : [LLaMa 7b](https://ai.facebook.com/blog/large-language-model-llama-meta-ai/) with CUDA acceleration implemented in rust. Minimal GPU memory needed!
- rustformers/llm
- Chidori
- Tokenizers - of-the-Art Tokenizers optimized for Research and Production. [huggingface.co/docs/tokenizers](https://huggingface.co/docs/tokenizers/index)
- dfdx
- luminal
- crabml
- TensorFlow Rust
- tch-rs - rs?style=social"/> : Rust bindings for the C++ api of PyTorch.
- rustai-solutions/candle_demo_openchat_35 - solutions/candle_demo_openchat_35?style=social"/> : candle_demo_openchat_35.
- llama2.rs
- Llama2-burn - burn?style=social"/> : Llama2 LLM ported to Rust burn.
- gaxler/llama2.rs
- whisper-burn - burn?style=social"/> : A Rust implementation of OpenAI's Whisper model using the burn framework.
- stable-diffusion-burn - diffusion-burn?style=social"/> : Stable Diffusion v1.4 ported to Rust's burn framework.
- coreylowman/llama-dfdx - dfdx?style=social"/> : [LLaMa 7b](https://ai.facebook.com/blog/large-language-model-llama-meta-ai/) with CUDA acceleration implemented in rust. Minimal GPU memory needed!
- tazz4843/whisper-rs - rs?style=social"/> : Rust bindings to [whisper.cpp](https://github.com/ggerganov/whisper.cpp).
- rustformers/llm
- Chidori
- llm-chain - chain?style=social"/> : llm-chain is a collection of Rust crates designed to help you work with Large Language Models (LLMs) more effectively. [llm-chain.xyz](https://llm-chain.xyz/)
- Atome-FE/llama-node - FE/llama-node?style=social"/> : Believe in AI democratization. llama for nodejs backed by llama-rs and llama.cpp, work locally on your laptop CPU. support llama/alpaca/gpt4all/vicuna model. [www.npmjs.com/package/llama-node](https://www.npmjs.com/package/llama-node)
- Noeda/rllama
- lencx/ChatGPT
- Synaptrix/ChatGPT-Desktop - Desktop?style=social"/> : Fuel your productivity with ChatGPT-Desktop - Blazingly fast and supercharged!
- Poordeveloper/chatgpt-app - app?style=social"/> : A ChatGPT App for all platforms. Built with Rust + Tauri + Vue + Axum.
- mxismean/chatgpt-app - app?style=social"/> : Tauri 项目：ChatGPT App.
- sonnylazuardi/chat-ai-desktop - ai-desktop?style=social"/> : Chat AI Desktop App. Unofficial ChatGPT desktop app for Mac & Windows menubar using Tauri & Rust.
- m1guelpf/browser-agent - agent?style=social"/> : A browser AI agent, using GPT-4. [docs.rs/browser-agent](https://docs.rs/browser-agent/latest/browser_agent/)
- sigoden/aichat - 3.5/GPT-4 in the terminal.
- uiuifree/rust-openai-chatgpt-api - openai-chatgpt-api?style=social"/> : "rust-openai-chatgpt-api" is a Rust library for accessing the ChatGPT API, a powerful NLP platform by OpenAI. The library provides a simple and efficient interface for sending requests and receiving responses, including chat. It uses reqwest and serde for HTTP requests and JSON serialization.
- 1595901624/gpt-aggregated-edition - aggregated-edition?style=social"/> : 聚合ChatGPT官方版、ChatGPT免费版、文心一言、Poe、chatchat等多平台，支持自定义导入平台。
- Cormanz/smartgpt
- femtoGPT
- shafishlabs/llmchain-rs - rs?style=social"/> : 🦀Rust + Large Language Models - Make AI Services Freely and Easily. Inspired by LangChain.
- flaneur2020/llama2.rs
- Heng30/chatbox - ui and Rust.
- fairjm/dioxus-openai-qa-gui - openai-qa-gui?style=social"/> : a simple openai qa desktop app built with dioxus.
- mxismean/chatgpt-app - app?style=social"/> : Tauri 项目：ChatGPT App.
- sonnylazuardi/chat-ai-desktop - ai-desktop?style=social"/> : Chat AI Desktop App. Unofficial ChatGPT desktop app for Mac & Windows menubar using Tauri & Rust.
- m1guelpf/browser-agent - agent?style=social"/> : A browser AI agent, using GPT-4. [docs.rs/browser-agent](https://docs.rs/browser-agent/latest/browser_agent/)
- llm-chain - chain?style=social"/> : llm-chain is a collection of Rust crates designed to help you work with Large Language Models (LLMs) more effectively. [llm-chain.xyz](https://llm-chain.xyz/)
- Atome-FE/llama-node - FE/llama-node?style=social"/> : Believe in AI democratization. llama for nodejs backed by llama-rs and llama.cpp, work locally on your laptop CPU. support llama/alpaca/gpt4all/vicuna model. [www.npmjs.com/package/llama-node](https://www.npmjs.com/package/llama-node)
- Noeda/rllama
- lencx/ChatGPT
- Synaptrix/ChatGPT-Desktop - Desktop?style=social"/> : Fuel your productivity with ChatGPT-Desktop - Blazingly fast and supercharged!
- Poordeveloper/chatgpt-app - app?style=social"/> : A ChatGPT App for all platforms. Built with Rust + Tauri + Vue + Axum.
- sigoden/aichat - 3.5/GPT-4 in the terminal.
- uiuifree/rust-openai-chatgpt-api - openai-chatgpt-api?style=social"/> : "rust-openai-chatgpt-api" is a Rust library for accessing the ChatGPT API, a powerful NLP platform by OpenAI. The library provides a simple and efficient interface for sending requests and receiving responses, including chat. It uses reqwest and serde for HTTP requests and JSON serialization.
- 1595901624/gpt-aggregated-edition - aggregated-edition?style=social"/> : 聚合ChatGPT官方版、ChatGPT免费版、文心一言、Poe、chatchat等多平台，支持自定义导入平台。
- Cormanz/smartgpt
- femtoGPT
- shafishlabs/llmchain-rs - rs?style=social"/> : 🦀Rust + Large Language Models - Make AI Services Freely and Easily. Inspired by LangChain.
- flaneur2020/llama2.rs
- Heng30/chatbox - ui and Rust.
- fairjm/dioxus-openai-qa-gui - openai-qa-gui?style=social"/> : a simple openai qa desktop app built with dioxus.
- llama2.zig
- llama2.zig
- renerocksai/gpt4all.zig - based chat client for an assistant-style large language model with ~800k GPT-3.5-Turbo Generations based on LLaMa.
- vLLM - project/vllm?style=social"/> : A high-throughput and memory-efficient inference and serving engine for LLMs. [docs.vllm.ai](https://docs.vllm.ai/)
- MLC LLM - ai/mlc-llm?style=social"/> : Enable everyone to develop, optimize and deploy AI models natively on everyone's devices. [mlc.ai/mlc-llm](https://mlc.ai/mlc-llm/)
- Lamini - ai/lamini?style=social"/> : Lamini: The LLM engine for rapidly customizing models 🦙.
- datawhalechina/self-llm - llm?style=social"/> : 《开源大模型食用指南》基于Linux环境快速部署开源大模型，更适合中国宝宝的部署教程。
- ninehills/llm-inference-benchmark - inference-benchmark?style=social"/> : LLM Inference benchmark.
- NVIDIA/nccl - GPU communication.
- NVIDIA/multi-gpu-programming-models - gpu-programming-models?style=social"/> : Examples demonstrating available options to program multiple GPUs in a single node or a cluster.
- wilicc/gpu-burn - burn?style=social"/> : Multi-GPU CUDA stress test.
- Cupoch - nat/cupoch?style=social"/> : Robotics with GPU computing.
- renerocksai/gpt4all.zig - based chat client for an assistant-style large language model with ~800k GPT-3.5-Turbo Generations based on LLaMa.
- vllm-project/vllm - project/vllm?style=social"/> : A high-throughput and memory-efficient inference and serving engine for LLMs. [vllm.readthedocs.io](https://vllm.readthedocs.io/en/latest/)
- MLC LLM - ai/mlc-llm?style=social"/> : Enable everyone to develop, optimize and deploy AI models natively on everyone's devices. [mlc.ai/mlc-llm](https://mlc.ai/mlc-llm/)
- Lamini - ai/lamini?style=social"/> : Lamini: The LLM engine for rapidly customizing models 🦙.
- datawhalechina/self-llm - llm?style=social"/> : 《开源大模型食用指南》基于Linux环境快速部署开源大模型，更适合中国宝宝的部署教程。
- ninehills/llm-inference-benchmark - inference-benchmark?style=social"/> : LLM Inference benchmark.
- NVIDIA/nccl - GPU communication.
- NVIDIA/multi-gpu-programming-models - gpu-programming-models?style=social"/> : Examples demonstrating available options to program multiple GPUs in a single node or a cluster.
- wilicc/gpu-burn - burn?style=social"/> : Multi-GPU CUDA stress test.
- Cupoch - nat/cupoch?style=social"/> : Robotics with GPU computing.
- Tachyon - network/tachyon?style=social"/> : Modular ZK(Zero Knowledge) backend accelerated by GPU.
- Blitzar - knowledge proof acceleration with GPUs for C++ and Rust. [www.spaceandtime.io/](https://www.spaceandtime.io/)
- blitzar-rs - rs?style=social"/> : High-Level Rust wrapper for the blitzar-sys crate. [www.spaceandtime.io/](https://www.spaceandtime.io/)
- ICICLE - zk/icicle?style=social"/> : ICICLE is a library for ZK acceleration using CUDA-enabled GPUs.
- Tachyon - network/tachyon?style=social"/> : Modular ZK(Zero Knowledge) backend accelerated by GPU.
- Blitzar - knowledge proof acceleration with GPUs for C++ and Rust. [www.spaceandtime.io/](https://www.spaceandtime.io/)
- blitzar-rs - rs?style=social"/> : High-Level Rust wrapper for the blitzar-sys crate. [www.spaceandtime.io/](https://www.spaceandtime.io/)
- ICICLE - zk/icicle?style=social"/> : ICICLE is a library for ZK acceleration using CUDA-enabled GPUs.
- YichengDWu/matmul.mojo - threaded implimentation of the [BLIS](https://en.wikipedia.org/wiki/BLIS_(software)) algorithm in pure Mojo 🔥.
- NVIDIA/cuda-python - python?style=social"/> : CUDA Python is the home for accessing NVIDIA’s CUDA platform from Python. CUDA Python Low-level Bindings. [nvidia.github.io/cuda-python/](https://nvidia.github.io/cuda-python/latest/)
- CuPy
- TensorRT - performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT. [developer.nvidia.com/tensorrt](https://developer.nvidia.com/tensorrt)
- TensorRT-LLM - LLM?style=social"/> : TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines. [nvidia.github.io/TensorRT-LLM](https://nvidia.github.io/TensorRT-LLM)
- linxihui/dkernel - 3-small models](https://huggingface.co/microsoft/Phi-3-small-8k-instruct). The sparse attention is also supported in vLLM for efficient inference.
- 'gpu' Dialect - level abstractions for launching GPU kernels following a programming model similar to that of CUDA or OpenCL.
- ONNX-MLIR - mlir?style=social"/> : Representation and Reference Lowering of ONNX Models in MLIR Compiler Infrastructure.
- TPU-MLIR - mlir?style=social"/> : Machine learning compiler based on MLIR for Sophgo TPU. TPU-MLIR is an open-source machine-learning compiler based on MLIR for TPU. This project provides a complete toolchain, which can convert pre-trained neural networks from different frameworks into binary files bmodel that can be efficiently operated on TPUs.
- IREE - org/iree?style=social"/> : IREE: Intermediate Representation Execution Environment. A retargetable MLIR-based machine learning compiler and runtime toolkit. [iree.dev/](http://iree.dev/)
- 'amdgpu' Dialect - specific functionality and LLVM intrinsics.
- pyMLIR - the Multi-Level Intermediate Representation. pyMLIR is a full Python interface to parse, process, and output [MLIR](https://mlir.llvm.org/) files according to the syntax described in the [MLIR documentation](https://github.com/llvm/llvm-project/tree/master/mlir/docs). pyMLIR supports the basic dialects and can be extended with other dialects.
- Torch-MLIR - mlir?style=social"/> : The Torch-MLIR project aims to provide first class support from the PyTorch ecosystem to the MLIR ecosystem.
- ByteIR - to-end model compilation solution. [byteir.ai](https://byteir.ai/)
- Xilinx/mlir-aie - aie?style=social"/> : An MLIR-based toolchain for AMD AI Engine-enabled devices. This repository contains an MLIR-based toolchain for AI Engine-enabled devices, such as [AMD Ryzen™ AI](https://www.amd.com/en/products/processors/consumer/ryzen-ai.html) and [Versal™](https://www.xilinx.com/products/technology/ai-engine.html).
- cuBLAS - accelerated library for accelerating AI and HPC applications. It includes several API extensions for providing drop-in industry standard BLAS APIs and GEMM APIs with support for fusions that are highly optimized for NVIDIA GPUs. The cuBLAS library also contains extensions for batched operations, execution across multiple GPUs, and mixed- and low-precision execution with additional tuning for the best performance.
- cuDNN - accelerated library of primitives for [deep neural networks](https://developer.nvidia.com/deep-learning). cuDNN provides highly tuned implementations for standard routines such as forward and backward convolution, attention, matmul, pooling, and normalization.
- zjhellofss/KuiperInfer (自制深度学习推理框架) - performance deep learning inference library step by step.
- YichengDWu/matmul.mojo - threaded implimentation of the [BLIS](https://en.wikipedia.org/wiki/BLIS_(software)) algorithm in pure Mojo 🔥.
- TensorRT - performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT. [developer.nvidia.com/tensorrt](https://developer.nvidia.com/tensorrt)
- TensorRT-LLM - LLM?style=social"/> : TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines. [nvidia.github.io/TensorRT-LLM](https://nvidia.github.io/TensorRT-LLM)
- DeployAI/nndeploy - zh.readthedocs.io/zh/latest/](https://nndeploy-zh.readthedocs.io/zh/latest/)
- purton-tech/bionicgpt - tech/bionicgpt?style=social"/> : Accelerate LLM adoption in your organisation. Chat with your confidential data safely and securely. [bionic-gpt.com](https://bionic-gpt.com/)
- EugenHotaj/zig_inference
- Ollama
- harleyszhang/lite_llama
- Liger-Kernel - Kernel?style=social"/> : Efficient Triton Kernels for LLM Training. [arxiv.org/pdf/2410.10989](https://arxiv.org/pdf/2410.10989)
- BobMcDear/attorch
- zjhellofss/KuiperLLama
- zjhellofss/kuiperdatawhale
- MarioSieg/magnetron
- lucasdelimanogueira/PyNorch - GPU support and automatic differentiation!)
- FlashAttention - AILab/flash-attention?style=social"/> : Fast and memory-efficient exact attention. "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness". (**[arXiv 2022](https://arxiv.org/abs/2205.14135)**).
- 66RING/tiny-flash-attention - flash-attention?style=social"/> : [flash attention](https://github.com/Dao-AILab/flash-attention) tutorial written in python, triton, cuda, cutlass.
- weishengying/tiny-flash-attention - flash-attention?style=social"/> : 使用 cutlass 实现 flash-attention 精简版，具有教学意义。
- jepeake/tiny-flash-attention - flash-attention?style=social"/> : flash attention in ~20 lines.
Official Version
- CUDA
- MLIR - Level Intermediate Representation Compiler Framework. The MLIR project is a novel approach to building reusable and extensible compiler infrastructure. MLIR aims to address software fragmentation, improve compilation for heterogeneous hardware, significantly reduce the cost of building domain specific compilers, and aid in connecting existing compilers together.
- TVM
- CUTLASS - performance matrix-matrix multiplication (GEMM) and related computations at all levels and scales within CUDA. It incorporates strategies for hierarchical decomposition and data movement similar to those used to implement [cuBLAS](https://developer.nvidia.com/cublas) and [cuDNN](https://developer.nvidia.com/cudnn).
- cuBLAS - accelerated library for accelerating AI and HPC applications. It includes several API extensions for providing drop-in industry standard BLAS APIs and GEMM APIs with support for fusions that are highly optimized for NVIDIA GPUs. The cuBLAS library also contains extensions for batched operations, execution across multiple GPUs, and mixed- and low-precision execution with additional tuning for the best performance.
- cuDNN - accelerated library of primitives for deep neural networks. cuDNN provides highly tuned implementations for standard routines such as forward and backward convolution, attention, matmul, pooling, and normalization.
Learning Resources
- xgqdut2016/cuda_code
- hyperai/triton-cn - cn?style=social"/> : Triton Documentation in Chinese Simplified / Triton 中文文档. [triton.hyper.ai](https://triton.hyper.ai/)
- Triton Docs
- xgqdut2016/hpc_project
- xgqdut2016/hpc2torch
- LAFF-On-PfHP - On Programming for High Performance.
- flame/how-to-optimize-gemm - to-optimize-gemm?style=social"/> : How To Optimize Gemm wiki pages. [https://github.com/flame/how-to-optimize-gemm/wiki](https://github.com/flame/how-to-optimize-gemm/wiki)
- flame/blislab
- tpoisonooo/how-to-optimize-gemm - to-optimize-gemm?style=social"/> : row-major matmul optimization. [zhuanlan.zhihu.com/p/65436463](https://zhuanlan.zhihu.com/p/65436463).
- NVIDIA CUDA Toolkit Documentation
- NVIDIA CUDA C++ Programming Guide
- NVIDIA CUDA C++ Best Practices Guide
- CuPy User Guide
- NVIDIA/cuda-samples - samples?style=social"/> : Samples for CUDA Developers which demonstrates features in CUDA Toolkit.
- NVIDIA/CUDALibrarySamples
- flame/how-to-optimize-gemm - to-optimize-gemm?style=social"/> : How To Optimize Gemm wiki pages. [https://github.com/flame/how-to-optimize-gemm/wiki](https://github.com/flame/how-to-optimize-gemm/wiki)
- flame/blislab
- tpoisonooo/how-to-optimize-gemm - to-optimize-gemm?style=social"/> : row-major matmul optimization. [zhuanlan.zhihu.com/p/65436463](https://zhuanlan.zhihu.com/p/65436463).
- LAFF-On-PfHP - On Programming for High Performance.
- NVIDIA CUDA Toolkit Documentation
- NVIDIA CUDA C++ Programming Guide
- NVIDIA CUDA C++ Best Practices Guide
- CuPy User Guide
- NVIDIA/cuda-samples - samples?style=social"/> : Samples for CUDA Developers which demonstrates features in CUDA Toolkit.
- NVIDIA/CUDALibrarySamples
- NVIDIA-developer-blog/code-samples - developer-blog/code-samples?style=social"/> : Source code examples from the [Parallel Forall Blog](http://developer.nvidia.com/parallel-forall).
- HeKun-NVIDIA/CUDA-Programming-Guide-in-Chinese - NVIDIA/CUDA-Programming-Guide-in-Chinese?style=social"/> : This is a Chinese translation of the CUDA programming guide. 本项目为 CUDA C Programming Guide 的中文翻译版。
- MAhaitao999/CUDA_Programming
- sangyc10/CUDA-code - code?style=social"/> : bilibili视频【CUDA编程基础入门系列（持续更新）】配套代码。
- NVIDIA-developer-blog/code-samples - developer-blog/code-samples?style=social"/> : Source code examples from the [Parallel Forall Blog](http://developer.nvidia.com/parallel-forall).
- HeKun-NVIDIA/CUDA-Programming-Guide-in-Chinese - NVIDIA/CUDA-Programming-Guide-in-Chinese?style=social"/> : This is a Chinese translation of the CUDA programming guide. 本项目为 CUDA C Programming Guide 的中文翻译版。
- cuda-mode/resource-stream - mode/resource-stream?style=social"/> : CUDA related news and material links.
- brucefan1983/CUDA-Programming - Programming?style=social"/> : Sample codes for my CUDA programming book.
- YouQixiaowu/CUDA-Programming-with-Python - Programming-with-Python?style=social"/> : 关于书籍CUDA Programming使用了pycuda模块的Python版本的示例代码。
- RussWong/CUDATutorial
- QINZHAOYU/CudaSteps - 基础与实践》（樊哲勇著）的cuda学习之路。
- DefTruth//CUDA-Learn-Notes - Learn-Notes?style=social"/> : 🎉CUDA/C++ 笔记 / 大模型手撕CUDA / 技术博客，更新随缘: flash_attn、sgemm、sgemv、warp reduce、block reduce、dot product、elementwise、softmax、layernorm、rmsnorm、hist etc.
- BBuf/how-to-optim-algorithm-in-cuda - to-optim-algorithm-in-cuda?style=social"/> : how to optimize some algorithm in cuda.
- PaddleJitLab/CUDATutorial - learning tutorail for CUDA High Performance Programing. 从零开始学习 CUDA 高性能编程。
- leimao/CUDA-GEMM-Optimization - GEMM-Optimization?style=social"/> : [CUDA Matrix Multiplication Optimization](https://leimao.github.io/article/CUDA-Matrix-Multiplication-Optimization/). This repository contains the CUDA kernels for general matrix-matrix multiplication (GEMM) and the corresponding performance analysis.
- interestingLSY/CUDA-From-Correctness-To-Performance-Code - From-Correctness-To-Performance-Code?style=social"/> : Codes & examples for "CUDA - From Correctness to Performance". The lecture can be found at [https://wiki.lcpu.dev/zh/hpc/from-scratch/cuda](https://wiki.lcpu.dev/zh/hpc/from-scratch/cuda).
- cuda-mode/resource-stream - mode/resource-stream?style=social"/> : CUDA related news and material links.
- brucefan1983/CUDA-Programming - Programming?style=social"/> : Sample codes for my CUDA programming book.
- QINZHAOYU/CudaSteps - 基础与实践》（樊哲勇著）的cuda学习之路。
- MAhaitao999/CUDA_Programming
- sangyc10/CUDA-code - code?style=social"/> : bilibili视频【CUDA编程基础入门系列（持续更新）】配套代码。
- RussWong/CUDATutorial
- DefTruth//CUDA-Learn-Notes - Learn-Notes?style=social"/> : 🎉CUDA/C++ 笔记 / 大模型手撕CUDA / 技术博客，更新随缘: flash_attn、sgemm、sgemv、warp reduce、block reduce、dot product、elementwise、softmax、layernorm、rmsnorm、hist etc.
- BBuf/how-to-optim-algorithm-in-cuda - to-optim-algorithm-in-cuda?style=social"/> : how to optimize some algorithm in cuda.
- PaddleJitLab/CUDATutorial - learning tutorail for CUDA High Performance Programing. 从零开始学习 CUDA 高性能编程。
- leimao/CUDA-GEMM-Optimization - GEMM-Optimization?style=social"/> : [CUDA Matrix Multiplication Optimization](https://leimao.github.io/article/CUDA-Matrix-Multiplication-Optimization/). This repository contains the CUDA kernels for general matrix-matrix multiplication (GEMM) and the corresponding performance analysis.
- interestingLSY/CUDA-From-Correctness-To-Performance-Code - From-Correctness-To-Performance-Code?style=social"/> : Codes & examples for "CUDA - From Correctness to Performance". The lecture can be found at [https://wiki.lcpu.dev/zh/hpc/from-scratch/cuda](https://wiki.lcpu.dev/zh/hpc/from-scratch/cuda).
- Liu-xiandong/How_to_optimize_in_GPU - xiandong/How_to_optimize_in_GPU?style=social"/> : This is a series of GPU optimization topics. Here we will introduce how to optimize the CUDA kernel in detail. I will introduce several basic kernel optimizations, including: elementwise, reduce, sgemv, sgemm, etc. The performance of these kernels is basically at or near the theoretical limit.
- Bruce-Lee-LY/matrix_multiply - Lee-LY/matrix_multiply?style=social"/> : Several common methods of matrix multiplication are implemented on CPU and Nvidia GPU using C++11 and CUDA.
- Bruce-Lee-LY/cuda_hgemm - Lee-LY/cuda_hgemm?style=social"/> : Several optimization methods of half-precision general matrix multiplication (HGEMM) using tensor core with WMMA API and MMA PTX instruction.
- Bruce-Lee-LY/cuda_hgemv - Lee-LY/cuda_hgemv?style=social"/> : Several optimization methods of half-precision general matrix vector multiplication (HGEMV) using CUDA core.
- enp1s0/ozIMMU
- AyakaGEMM/Hands-on-GEMM - on-GEMM?style=social"/> : A GEMM tutorial.
- Cjkkkk/CUDA_gemm
- AyakaGEMM/Hands-on-MLIR - on-MLIR?style=social"/> : Hands-on-MLIR.
- zpzim/MSplitGEMM
- jundaf2/CUDA-INT8-GEMM - INT8-GEMM?style=social"/> : CUDA 8-bit Tensor Core Matrix Multiplication based on m16n16k16 WMMA API.
- chanzhennan/cuda_gemm_benchmark - xiandong/How_to_optimize_in_GPU](https://github.com/Liu-xiandong/How_to_optimize_in_GPU).
- YuxueYang1204/CudaDemo
- CoffeeBeforeArch/cuda_programming
- rbaygildin/learn-gpgpu - gpgpu?style=social"/> : Algorithms implemented in CUDA + resources about GPGPU.
- godweiyang/NN-CUDA-Example - CUDA-Example?style=social"/> : Several simple examples for popular neural network toolkits calling custom CUDA operators.
- Liu-xiandong/How_to_optimize_in_GPU - xiandong/How_to_optimize_in_GPU?style=social"/> : This is a series of GPU optimization topics. Here we will introduce how to optimize the CUDA kernel in detail. I will introduce several basic kernel optimizations, including: elementwise, reduce, sgemv, sgemm, etc. The performance of these kernels is basically at or near the theoretical limit.
- Bruce-Lee-LY/matrix_multiply - Lee-LY/matrix_multiply?style=social"/> : Several common methods of matrix multiplication are implemented on CPU and Nvidia GPU using C++11 and CUDA.
- Bruce-Lee-LY/cuda_hgemm - Lee-LY/cuda_hgemm?style=social"/> : Several optimization methods of half-precision general matrix multiplication (HGEMM) using tensor core with WMMA API and MMA PTX instruction.
- Bruce-Lee-LY/cuda_hgemv - Lee-LY/cuda_hgemv?style=social"/> : Several optimization methods of half-precision general matrix vector multiplication (HGEMV) using CUDA core.
- enp1s0/ozIMMU
- Cjkkkk/CUDA_gemm
- AyakaGEMM/Hands-on-GEMM - on-GEMM?style=social"/> : A GEMM tutorial.
- AyakaGEMM/Hands-on-MLIR - on-MLIR?style=social"/> : Hands-on-MLIR.
- zpzim/MSplitGEMM
- jundaf2/CUDA-INT8-GEMM - INT8-GEMM?style=social"/> : CUDA 8-bit Tensor Core Matrix Multiplication based on m16n16k16 WMMA API.
- chanzhennan/cuda_gemm_benchmark - xiandong/How_to_optimize_in_GPU](https://github.com/Liu-xiandong/How_to_optimize_in_GPU).
- YuxueYang1204/CudaDemo
- CoffeeBeforeArch/cuda_programming
- rbaygildin/learn-gpgpu - gpgpu?style=social"/> : Algorithms implemented in CUDA + resources about GPGPU.
- godweiyang/NN-CUDA-Example - CUDA-Example?style=social"/> : Several simple examples for popular neural network toolkits calling custom CUDA operators.
- yhwang-hub/Matrix_Multiplication_Performance_Optimization - hub/Matrix_Multiplication_Performance_Optimization?style=social"/> : Matrix Multiplication Performance Optimization.
- yao-jiashu/KernelCodeGen - jiashu/KernelCodeGen?style=social"/> : GEMM/Conv2d CUDA/HIP kernel code generation using MLIR.
- caiwanxianhust/ClusteringByCUDA
- ulrichstern/cuda-convnet - convnet?style=social"/> : Alex Krizhevsky's original code from Google Code. "微信公众号「人工智能大讲堂」《[找到了AlexNet当年的源代码，没用框架，从零手撸CUDA/C++](https://mp.weixin.qq.com/s/plxXG8y5QlxSionyjyPXqw)》"。
- PacktPublishing/Learn-CUDA-Programming - CUDA-Programming?style=social"/> : Learn CUDA Programming, published by Packt.
- PacktPublishing/Hands-On-GPU-Programming-with-Python-and-CUDA - On-GPU-Programming-with-Python-and-CUDA?style=social"/> : Hands-On GPU Programming with Python and CUDA, published by Packt.
- PacktPublishing/Hands-On-GPU-Accelerated-Computer-Vision-with-OpenCV-and-CUDA - On-GPU-Accelerated-Computer-Vision-with-OpenCV-and-CUDA?style=social"/> : Hands-On GPU Accelerated Computer Vision with OpenCV and CUDA, published by Packt.
- codingonion/cuda-beginner-course-cpp-version - beginner-course-cpp-version?style=social"/> : bilibili视频【CUDA 12.x 并行编程入门(C++版)】配套代码。
- codingonion/cuda-beginner-course-python-version - beginner-course-python-version?style=social"/> : bilibili视频【CUDA 12.x 并行编程入门(Python版)】配套代码。
- codingonion/cuda-beginner-course-rust-version - beginner-course-rust-version?style=social"/> : bilibili视频【CUDA 12.x 并行编程入门(Rust版)】配套代码。
- NVIDIA TensorRT Docs
- HeKun-NVIDIA/TensorRT-Developer_Guide_in_Chinese - NVIDIA/TensorRT-Developer_Guide_in_Chinese?style=social"/> : 本项目是NVIDIA TensorRT的中文版开发手册，有个人翻译并添加自己的理解。
- LitLeo/TensorRT_Tutorial
- yao-jiashu/KernelCodeGen - jiashu/KernelCodeGen?style=social"/> : GEMM/Conv2d CUDA/HIP kernel code generation using MLIR.
- caiwanxianhust/ClusteringByCUDA
- ulrichstern/cuda-convnet - convnet?style=social"/> : Alex Krizhevsky's original code from Google Code. "微信公众号「人工智能大讲堂」《[找到了AlexNet当年的源代码，没用框架，从零手撸CUDA/C++](https://mp.weixin.qq.com/s/plxXG8y5QlxSionyjyPXqw)》"。
- PacktPublishing/Learn-CUDA-Programming - CUDA-Programming?style=social"/> : Learn CUDA Programming, published by Packt.
- PacktPublishing/Hands-On-GPU-Programming-with-Python-and-CUDA - On-GPU-Programming-with-Python-and-CUDA?style=social"/> : Hands-On GPU Programming with Python and CUDA, published by Packt.
- PacktPublishing/Hands-On-GPU-Accelerated-Computer-Vision-with-OpenCV-and-CUDA - On-GPU-Accelerated-Computer-Vision-with-OpenCV-and-CUDA?style=social"/> : Hands-On GPU Accelerated Computer Vision with OpenCV and CUDA, published by Packt.
- codingonion/cuda-beginner-course-cpp-version - beginner-course-cpp-version?style=social"/> : bilibili视频【CUDA 12.x 并行编程入门(C++版)】配套代码。
- codingonion/cuda-beginner-course-python-version - beginner-course-python-version?style=social"/> : bilibili视频【CUDA 12.x 并行编程入门(Python版)】配套代码。
- codingonion/cuda-beginner-course-rust-version - beginner-course-rust-version?style=social"/> : bilibili视频【CUDA 12.x 并行编程入门(Rust版)】配套代码。
- NVIDIA TensorRT Docs
- HeKun-NVIDIA/TensorRT-Developer_Guide_in_Chinese - NVIDIA/TensorRT-Developer_Guide_in_Chinese?style=social"/> : 本项目是NVIDIA TensorRT的中文版开发手册，有个人翻译并添加自己的理解。
- LitLeo/TensorRT_Tutorial
- ifromeast/cuda_learning
- LLVM Docs
- MLIR Docs
- BBuf/tvm_mlir_learn
- j2kun/mlir-tutorial - tutorial?style=social"/> : This is the code repository for a series of articles on the [MLIR framework](https://mlir.llvm.org/) for building compilers.
- KEKE046/mlir-tutorial - tutorial?style=social"/> : Hands-On Practical MLIR Tutorial.
- Triton - lang/triton?style=social"/> : Development repository for the Triton language and compiler. [triton-lang.org/](https://triton-lang.org/)
- chenzomi12/AISystem
- chenzomi12/AIFoundation
- CuPy
- BobMcDear/neural-network-cuda - network-cuda?style=social"/> : Neural network from scratch in CUDA/C++.
- Apache TVM 中文站
Awesome List
- Erkaman/Awesome-CUDA - CUDA?style=social"/> : This is a list of useful libraries and resources for CUDA development.
- jslee02/awesome-gpgpu - gpgpu?style=social"/> : 😎 A curated list of awesome GPGPU (CUDA/OpenCL/Vulkan) resources.
- mikeroyal/CUDA-Guide - Guide?style=social"/> : A guide covering CUDA including the applications and tools that will make you a better and more efficient CUDA developer.
- Erkaman/Awesome-CUDA - CUDA?style=social"/> : This is a list of useful libraries and resources for CUDA development.
- jslee02/awesome-gpgpu - gpgpu?style=social"/> : 😎 A curated list of awesome GPGPU (CUDA/OpenCL/Vulkan) resources.
- mikeroyal/CUDA-Guide - Guide?style=social"/> : A guide covering CUDA including the applications and tools that will make you a better and more efficient CUDA developer.
- rkinas/triton-resources - resources?style=social"/> : A curated list of resources for learning and exploring Triton, OpenAI's programming language for writing efficient GPU code.
Jobs and Interview
Applications
- emptysoal/cuda-image-preprocess - image-preprocess?style=social"/> : Speed up image preprocess with cuda when handle image or tensorrt inference. Cuda编程加速图像预处理。
- Melody-Zhou/tensorRT_Pro-YOLOv8 - Zhou/tensorRT_Pro-YOLOv8?style=social"/> : This repository is based on [shouxieai/tensorRT_Pro](https://github.com/shouxieai/tensorRT_Pro), with adjustments to support YOLOv8. 目前已支持 YOLOv8、YOLOv8-Cls、YOLOv8-Seg、YOLOv8-OBB、YOLOv8-Pose、RT-DETR、ByteTrack、YOLOv9、YOLOv10、RTMO 高性能推理！！！🚀🚀🚀
- shouxieai/tensorRT_Pro
- shouxieai/infer
- hamdiboukamcha/yolov10-tensorrt - tensorrt?style=social"/> : YOLOv10 C++ TensorRT : Real-Time End-to-End Object Detection.
- triple-Mu/YOLOv8-TensorRT - Mu/YOLOv8-TensorRT?style=social"/> : YOLOv8 using TensorRT accelerate !
- emptysoal/cuda-image-preprocess - image-preprocess?style=social"/> : Speed up image preprocess with cuda when handle image or tensorrt inference. Cuda编程加速图像预处理。
- laugh12321/TensorRT-YOLO - YOLO?style=social"/> : 🚀 TensorRT-YOLO: Support YOLOv3, YOLOv5, YOLOv6, YOLOv7, YOLOv8, YOLOv9, YOLOv10, PP-YOLOE using TensorRT acceleration with EfficientNMS! TensorRT-YOLO 是一个支持 YOLOv3、YOLOv5、YOLOv6、YOLOv7、YOLOv8、YOLOv9、YOLOv10、PP-YOLOE 和 PP-YOLOE+ 的推理加速项目，使用 NVIDIA TensorRT 进行优化。项目不仅集成了 EfficientNMS TensorRT 插件以增强后处理效果，还使用了 CUDA 核函数来加速前处理过程。TensorRT-YOLO 提供了 C++ 和 Python 推理的支持，旨在提供快速而优化的目标检测解决方案。
- l-sf/Linfer - sf/Linfer?style=social"/> : 基于TensorRT的C++高性能推理库，Yolov10, YoloPv2，Yolov5/7/X/8，RT-DETR，单目标跟踪OSTrack、LightTrack。
- FeiYull/TensorRT-Alpha - AI-IOT/torch2trt?style=social"/> : 🔥🔥🔥TensorRT for YOLOv8、YOLOv8-Pose、YOLOv8-Seg、YOLOv8-Cls、YOLOv7、YOLOv6、YOLOv5、YOLONAS......🚀🚀🚀CUDA IS ALL YOU NEED.🍎🍎🍎
- laugh12321/TensorRT-YOLO - YOLO?style=social"/> : 🚀 TensorRT-YOLO: Support YOLOv3, YOLOv5, YOLOv6, YOLOv7, YOLOv8, YOLOv9, YOLOv10, PP-YOLOE using TensorRT acceleration with EfficientNMS! TensorRT-YOLO 是一个支持 YOLOv3、YOLOv5、YOLOv6、YOLOv7、YOLOv8、YOLOv9、YOLOv10、PP-YOLOE 和 PP-YOLOE+ 的推理加速项目，使用 NVIDIA TensorRT 进行优化。项目不仅集成了 EfficientNMS TensorRT 插件以增强后处理效果，还使用了 CUDA 核函数来加速前处理过程。TensorRT-YOLO 提供了 C++ 和 Python 推理的支持，旨在提供快速而优化的目标检测解决方案。
- l-sf/Linfer - sf/Linfer?style=social"/> : 基于TensorRT的C++高性能推理库，Yolov10, YoloPv2，Yolov5/7/X/8，RT-DETR，单目标跟踪OSTrack、LightTrack。
- Melody-Zhou/tensorRT_Pro-YOLOv8 - Zhou/tensorRT_Pro-YOLOv8?style=social"/> : This repository is based on [shouxieai/tensorRT_Pro](https://github.com/shouxieai/tensorRT_Pro), with adjustments to support YOLOv8. 目前已支持 YOLOv8、YOLOv8-Cls、YOLOv8-Seg、YOLOv8-OBB、YOLOv8-Pose、RT-DETR、ByteTrack、YOLOv9、YOLOv10、RTMO 高性能推理！！！🚀🚀🚀
- shouxieai/tensorRT_Pro
- shouxieai/infer
- hamdiboukamcha/yolov10-tensorrt - tensorrt?style=social"/> : YOLOv10 C++ TensorRT : Real-Time End-to-End Object Detection.
- triple-Mu/YOLOv8-TensorRT - Mu/YOLOv8-TensorRT?style=social"/> : YOLOv8 using TensorRT accelerate !
- FeiYull/TensorRT-Alpha - AI-IOT/torch2trt?style=social"/> : 🔥🔥🔥TensorRT for YOLOv8、YOLOv8-Pose、YOLOv8-Seg、YOLOv8-Cls、YOLOv7、YOLOv6、YOLOv5、YOLONAS......🚀🚀🚀CUDA IS ALL YOU NEED.🍎🍎🍎
- cyrusbehr/YOLOv8-TensorRT-CPP - TensorRT-CPP?style=social"/> : YOLOv8 TensorRT C++ Implementation. A C++ Implementation of YoloV8 using TensorRT Supports object detection, semantic segmentation, and body pose estimation.
- VIDIA-AI-IOT/torch2trt - AI-IOT/torch2trt?style=social"/> : An easy to use PyTorch to TensorRT converter.
- zhiqwang/yolort
- Linaom1214/TensorRT-For-YOLO-Series - For-YOLO-Series?style=social"/> : YOLO Series TensorRT Python/C++. tensorrt for yolo series (YOLOv8, YOLOv7, YOLOv6....), nms plugin support.
- wang-xinyu/tensorrtx - xinyu/tensorrtx?style=social"/> : TensorRTx aims to implement popular deep learning networks with tensorrt network definition APIs.
- cyrusbehr/YOLOv8-TensorRT-CPP - TensorRT-CPP?style=social"/> : YOLOv8 TensorRT C++ Implementation. A C++ Implementation of YoloV8 using TensorRT Supports object detection, semantic segmentation, and body pose estimation.
- VIDIA-AI-IOT/torch2trt - AI-IOT/torch2trt?style=social"/> : An easy to use PyTorch to TensorRT converter.
- zhiqwang/yolort
- Linaom1214/TensorRT-For-YOLO-Series - For-YOLO-Series?style=social"/> : YOLO Series TensorRT Python/C++. tensorrt for yolo series (YOLOv8, YOLOv7, YOLOv6....), nms plugin support.
- wang-xinyu/tensorrtx - xinyu/tensorrtx?style=social"/> : TensorRTx aims to implement popular deep learning networks with tensorrt network definition APIs.
- DefTruth/lite.ai.toolkit
- PaddlePaddle/FastDeploy - to-use and Fast Deep Learning Model Deployment Toolkit for ☁️Cloud 📱Mobile and 📹Edge. Including Image, Video, Text and Audio 20+ main stream scenarios and 150+ SOTA models with end-to-end optimization, multi-platform and multi-framework support.
- enazoe/yolo-tensorrt - tensorrt?style=social"/> : TensorRT8.Support Yolov5n,s,m,l,x .darknet -> tensorrt. Yolov4 Yolov3 use raw darknet *.weights and *.cfg fils. If the wrapper is useful to you,please Star it.
- guojianyang/cv-detect-robot - detect-robot?style=social"/> : 🔥🔥🔥🔥🔥🔥Docker NVIDIA Docker2 YOLOV5 YOLOX YOLO Deepsort TensorRT ROS Deepstream Jetson Nano TX2 NX for High-performance deployment(高性能部署)。
- BlueMirrors/Yolov5-TensorRT - TensorRT?style=social"/> : Yolov5 TensorRT Implementations.
- lewes6369/TensorRT-Yolov3 - Yolov3?style=social"/> : TensorRT for Yolov3.
- CaoWGG/TensorRT-YOLOv4 - YOLOv4?style=social"/> :tensorrt5, yolov4, yolov3,yolov3-tniy,yolov3-tniy-prn.
- isarsoft/yolov4-triton-tensorrt - triton-tensorrt?style=social"/> : YOLOv4 on Triton Inference Server with TensorRT.
- TrojanXu/yolov5-tensorrt - tensorrt?style=social"/> : A tensorrt implementation of yolov5.
- tjuskyzhang/Scaled-YOLOv4-TensorRT - YOLOv4-TensorRT?style=social"/> : Implement yolov4-tiny-tensorrt, yolov4-csp-tensorrt, yolov4-large-tensorrt(p5, p6, p7) layer by layer using TensorRT API.
- Syencil/tensorRT - 7 Network Lib 包括常用目标检测、关键点检测、人脸检测、OCR等可训练自己数据。
- SeanAvery/yolov5-tensorrt - tensorrt?style=social"/> : YOLOv5 in TensorRT.
- Monday-Leo/YOLOv7_Tensorrt - Leo/YOLOv7_Tensorrt?style=social"/> : A simple implementation of Tensorrt YOLOv7.
- ibaiGorordo/ONNX-YOLOv6-Object-Detection - YOLOv6-Object-Detection?style=social"/> : Python scripts performing object detection using the YOLOv6 model in ONNX.
- ibaiGorordo/ONNX-YOLOv7-Object-Detection - YOLOv7-Object-Detection?style=social"/> : Python scripts performing object detection using the YOLOv7 model in ONNX.
- triple-Mu/yolov7 - Mu/yolov7?style=social"/> : End2end TensorRT YOLOv7.
- hewen0901/yolov7_trt
- tsutof/tiny_yolov2_onnx_cam
- Monday-Leo/Yolov5_Tensorrt_Win10 - Leo/Yolov5_Tensorrt_Win10?style=social"/> : A simple implementation of tensorrt yolov5 python/c++🔥
- Wulingtian/yolov5_tensorrt_int8
- Wulingtian/yolov5_tensorrt_int8_tools
- MadaoFY/yolov5_TensorRT_inference
- DefTruth/lite.ai.toolkit
- PaddlePaddle/FastDeploy - to-use and Fast Deep Learning Model Deployment Toolkit for ☁️Cloud 📱Mobile and 📹Edge. Including Image, Video, Text and Audio 20+ main stream scenarios and 150+ SOTA models with end-to-end optimization, multi-platform and multi-framework support.
- enazoe/yolo-tensorrt - tensorrt?style=social"/> : TensorRT8.Support Yolov5n,s,m,l,x .darknet -> tensorrt. Yolov4 Yolov3 use raw darknet *.weights and *.cfg fils. If the wrapper is useful to you,please Star it.
- guojianyang/cv-detect-robot - detect-robot?style=social"/> : 🔥🔥🔥🔥🔥🔥Docker NVIDIA Docker2 YOLOV5 YOLOX YOLO Deepsort TensorRT ROS Deepstream Jetson Nano TX2 NX for High-performance deployment(高性能部署)。
- BlueMirrors/Yolov5-TensorRT - TensorRT?style=social"/> : Yolov5 TensorRT Implementations.
- lewes6369/TensorRT-Yolov3 - Yolov3?style=social"/> : TensorRT for Yolov3.
- CaoWGG/TensorRT-YOLOv4 - YOLOv4?style=social"/> :tensorrt5, yolov4, yolov3,yolov3-tniy,yolov3-tniy-prn.
- isarsoft/yolov4-triton-tensorrt - triton-tensorrt?style=social"/> : YOLOv4 on Triton Inference Server with TensorRT.
- TrojanXu/yolov5-tensorrt - tensorrt?style=social"/> : A tensorrt implementation of yolov5.
- tjuskyzhang/Scaled-YOLOv4-TensorRT - YOLOv4-TensorRT?style=social"/> : Implement yolov4-tiny-tensorrt, yolov4-csp-tensorrt, yolov4-large-tensorrt(p5, p6, p7) layer by layer using TensorRT API.
- SeanAvery/yolov5-tensorrt - tensorrt?style=social"/> : YOLOv5 in TensorRT.
- Syencil/tensorRT - 7 Network Lib 包括常用目标检测、关键点检测、人脸检测、OCR等可训练自己数据。
- Monday-Leo/YOLOv7_Tensorrt - Leo/YOLOv7_Tensorrt?style=social"/> : A simple implementation of Tensorrt YOLOv7.
- ibaiGorordo/ONNX-YOLOv6-Object-Detection - YOLOv6-Object-Detection?style=social"/> : Python scripts performing object detection using the YOLOv6 model in ONNX.
- ibaiGorordo/ONNX-YOLOv7-Object-Detection - YOLOv7-Object-Detection?style=social"/> : Python scripts performing object detection using the YOLOv7 model in ONNX.
- triple-Mu/yolov7 - Mu/yolov7?style=social"/> : End2end TensorRT YOLOv7.
- hewen0901/yolov7_trt
- tsutof/tiny_yolov2_onnx_cam
- tatsuya-fukuoka/yolov7-onnx-infer - fukuoka/yolov7-onnx-infer?style=social"/> : Inference with yolov7's onnx model.
- Monday-Leo/Yolov5_Tensorrt_Win10 - Leo/Yolov5_Tensorrt_Win10?style=social"/> : A simple implementation of tensorrt yolov5 python/c++🔥
- ervgan/yolov5_tensorrt_inference
- Wulingtian/yolov5_tensorrt_int8
- Wulingtian/yolov5_tensorrt_int8_tools
- MadaoFY/yolov5_TensorRT_inference
- ibaiGorordo/ONNX-YOLOv8-Object-Detection - YOLOv8-Object-Detection?style=social"/> : Python scripts performing object detection using the YOLOv8 model in ONNX.
- we0091234/yolov8-tensorrt - tensorrt?style=social"/> : yolov8 tensorrt 加速.
- FeiYull/yolov8-tensorrt - tensorrt?style=social"/> : YOLOv8的TensorRT+CUDA加速部署，代码可在Win、Linux下运行。
- cvdong/YOLO_TRT_SIM
- cvdong/YOLO_TRT_PY
- Psynosaur/Jetson-SecVision - SecVision?style=social"/> : Person detection for Hikvision DVR with AlarmIO ports, uses TensorRT and yolov4.
- AlbinZhu/easy-trt - trt?style=social"/> : TensorRT for YOLOv10 with CUDA.
- cvdong/YOLO_TRT_SIM
- cvdong/YOLO_TRT_PY
- ibaiGorordo/ONNX-YOLOv8-Object-Detection - YOLOv8-Object-Detection?style=social"/> : Python scripts performing object detection using the YOLOv8 model in ONNX.
- we0091234/yolov8-tensorrt - tensorrt?style=social"/> : yolov8 tensorrt 加速.
- Psynosaur/Jetson-SecVision - SecVision?style=social"/> : Person detection for Hikvision DVR with AlarmIO ports, uses TensorRT and yolov4.
- FeiYull/yolov8-tensorrt - tensorrt?style=social"/> : YOLOv8的TensorRT+CUDA加速部署，代码可在Win、Linux下运行。
- tatsuya-fukuoka/yolov7-onnx-infer - fukuoka/yolov7-onnx-infer?style=social"/> : Inference with yolov7's onnx model.
- ervgan/yolov5_tensorrt_inference
- AlbinZhu/easy-trt - trt?style=social"/> : TensorRT for YOLOv10 with CUDA.
- kalfazed/tensorrt_starter
- kalfazed/tensorrt_starter

Programming Languages

Rust 135 C++ 127 Python 78 Cuda 68 C 16 Jupyter Notebook 7 Vue 6 Mojo 6 Julia 6 HTML 4

Ecosyste.ms: Awesome

awesome-cuda-triton-hpc

Blogs

Frameworks

Official Version

Learning Resources

Awesome List

Jobs and Interview

Applications