{"id":13809600,"url":"https://github.com/Liu-xiandong/How_to_optimize_in_GPU","last_synced_at":"2025-05-14T08:32:52.133Z","repository":{"id":37447030,"uuid":"418155000","full_name":"Liu-xiandong/How_to_optimize_in_GPU","owner":"Liu-xiandong","description":"This is a series of GPU optimization topics. Here we will introduce  how to optimize the CUDA kernel in detail.  I will introduce several basic kernel optimizations, including: elementwise, reduce, sgemv, sgemm, etc. The performance of these kernels is basically at or near the theoretical limit.","archived":false,"fork":false,"pushed_at":"2023-07-29T05:52:37.000Z","size":1310,"stargazers_count":825,"open_issues_count":6,"forks_count":131,"subscribers_count":13,"default_branch":"master","last_synced_at":"2024-11-13T19:45:36.165Z","etag":null,"topics":["elementwise","gpu-acceleration","high-performance-computing","hpc","reduce","sgemm","sgemv"],"latest_commit_sha":null,"homepage":"","language":"Cuda","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Liu-xiandong.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-17T14:19:27.000Z","updated_at":"2024-11-13T14:52:17.000Z","dependencies_parsed_at":"2024-08-04T02:01:33.848Z","dependency_job_id":null,"html_url":"https://github.com/Liu-xiandong/How_to_optimize_in_GPU","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Liu-xiandong%2FHow_to_optimize_in_GPU","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Liu-xiandong%2FHow_to_optimize_in_GPU/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Liu-xiandong%2FHow_to_optimize_in_GPU/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Liu-xiandong%2FHow_to_optimize_in_GPU/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Liu-xiandong","download_url":"https://codeload.github.com/Liu-xiandong/How_to_optimize_in_GPU/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225282502,"owners_count":17449524,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elementwise","gpu-acceleration","high-performance-computing","hpc","reduce","sgemm","sgemv"],"created_at":"2024-08-04T02:00:32.142Z","updated_at":"2024-11-19T02:30:57.936Z","avatar_url":"https://github.com/Liu-xiandong.png","language":"Cuda","funding_links":[],"categories":["Learning Resources"],"sub_categories":[],"readme":"# How to optimize in GPU\nThis is a series of GPU optimization topics. Here we will introduce  how to optimize the program on the GPU in detail. Here, I will introduce several basic kernel optimizations, including: elementwise, reduce, sgemv, sgemm, etc. All the following performance data are run on V100 and tested by nsight. \n\nIf you have any questions, you can directly contact: xiandong_liu@foxmail.com\n\n## 1. elementwise\nFor elementwise kernel, the optimization techniques that can be used are mainly vectorized memory access. I compared the three memory access methods of float, float2, and float4. The performance is as follows:\n\n| Type  |  Bandwiths | Ratio  |\n| :------------: | :------------: | :------------: |\n|  float | 827  | 91.9%  |\n|  float2 | 838  | 93.1%  |\n| float4  | 844  | 93.8%  |\n\n\n## 2. reduce\nSeven optimization methods were used to optimize reduce operator, and the performance of different optimization methods was tested on V100. The bandwidth of **858GB/s** is obtained, and the bandwidth utilization rate is: **858/900=95.3%**. The figure below shows the performance of the 7 optimization techniques.\n![](https://github.com/Liu-xiandong/How_to_optimize_in_GPU/blob/master/figure/reduce.png?raw=true)\n\n## 3. sgemv\nThe core of sgemv kernel optimization lies in the design of blocks and threads, and it is necessary to avoid the situation of thread idle as much as possible.\n\nThere are different optimization techniques for different data shapes. I designed 3 different optimization methods for 3 different situations, namely: n=32, n\u003c32, and n\u003e32. The corresponding performance is as follows:\n\n| sgemv  | M  | N  |  my_sgemv time(ns) | cublas(ns)  | my_sgemv/cublas  |\n| :------------: | :------------: | :------------: | :------------: | :------------: | :------------: |\n| v0  | 16384  | 32  | 10341  |  8386 | 81.1%  |\n| v1  | 16384  |  128 | 14284  | 15848  |  110.9% |\n| v2  | 16384  | 16  | 6903  | 7576  |  109.7% |\n\n## 4. sgemm\nThe optimization of sgemm is divided into two levels, namely CUDA\nC-level optimization and optimization of SASS code.\n\nRegarding CUDA C-level optimizations, the final code is sgemm_v3. On a large matrix of 4096 (M=N=K), our sgemm can achieve **96.8%** performance of cublas, with a peak floating point efficiency of 93.6%, basically reaching the limit of CUDA C code optimization. The figure below shows the performance of sgemm_v3.\n![](https://github.com/Liu-xiandong/How_to_optimize_in_GPU/blob/master/figure/sgemm.png?raw=true)\n\nRegarding the optimization of SASS code, CuAssembler is used for tuning. There are two main optimization techniques in this part, namely register remapping and instruction rearrangement to obtain better .reuse flag layout.\n\n## License\nAll the source codes of this repo are released under [Apache 2.0](http://www.apache.org/licenses/LICENSE-2.0).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FLiu-xiandong%2FHow_to_optimize_in_GPU","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FLiu-xiandong%2FHow_to_optimize_in_GPU","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FLiu-xiandong%2FHow_to_optimize_in_GPU/lists"}