https://github.com/vikparuchuri/triton_tutorial

Tutorials for Triton, a language for writing gpu kernels
https://github.com/vikparuchuri/triton_tutorial

Last synced: about 1 month ago
JSON representation

Tutorials for Triton, a language for writing gpu kernels

• Host: GitHub
• URL: https://github.com/vikparuchuri/triton_tutorial
• Owner: VikParuchuri
• Created: 2023-08-21T19:26:05.000Z (about 1 year ago)
• Default Branch: master
• Last Pushed: 2023-08-23T18:24:58.000Z (about 1 year ago)
• Last Synced: 2024-04-16T06:57:05.097Z (7 months ago)
• Language: Jupyter Notebook
• Homepage:
• Size: 1.07 MB
• Stars: 4
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Triton tutorials

[Triton](https://www.github.com/openai/triton) is a language for writing GPU kernels.  It's easier to use than CUDA, and interoperates well with PyTorch.

If you want to speed up PyTorch training or inference speed, you can try writing kernels for the heavier operations using Triton. ([flash attention](https://github.com/Dao-AILab/flash-attention) is a good example of a custom GPU kernel that speeds up training)

This repo has my notes as I learn to use Triton.  They include a lot of code, and some discussion of the key concepts.  They're geared towards people new to GPU programming and Triton.

Hopefully you will find them useful.

# Contents

1. [GPU Basics](01_gpu_basics.ipynb)

2. [Vector Addition](02_vector_addition.ipynb)

3. [Matrix Multiplication](03_small_matrix_multiplication.ipynb)

4. [Softmax forward and backward](04_softmax_fwd_bwd.ipynb)

5. [Block matmul](05_block_matmul.ipynb)

6. [Matmul forward and backward](06_matmul_fwd_bwd.ipynb)

# Install

To install Triton, just do `pip install triton`.  You need a CUDA-compatible GPU with CUDA installed to use it.

# References

Material in these notebooks came from the following sources (and they're generally good documentation):

- [Triton tutorials](https://triton-lang.org/main/index.html)

- [NVIDIA CUDA guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html)

- [CUDA MMM](https://siboehm.com/articles/22/CUDA-MMM)