https://github.com/hsm207/howto-llamacpp-with-gpu

Step-by-step guide on running LLaMA language models using llama.cpp with GPU acceleration. Includes detailed examples and performance comparison. Based on OpenLLaMA project.
https://github.com/hsm207/howto-llamacpp-with-gpu

artificial-intelligence deep-learning generativeai llms natural-language-processing

Last synced: 5 months ago
JSON representation

Step-by-step guide on running LLaMA language models using llama.cpp with GPU acceleration. Includes detailed examples and performance comparison. Based on OpenLLaMA project.

• Host: GitHub
• URL: https://github.com/hsm207/howto-llamacpp-with-gpu
• Owner: hsm207
• Created: 2023-05-30T11:16:15.000Z (about 3 years ago)
• Default Branch: main
• Last Pushed: 2023-05-30T21:08:07.000Z (about 3 years ago)
• Last Synced: 2025-10-08T10:30:54.517Z (9 months ago)
• Topics: artificial-intelligence, deep-learning, generativeai, llms, natural-language-processing
• Language: Makefile
• Homepage:
• Size: 4.88 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# Introduction

This is the code for the [The Total Noob's Guide to Harnessing the GPU for LLaMA Inference](https://medium.com/@_init_/the-total-noobs-guide-to-harnessing-the-gpu-for-llama-inference-2e3c4fdfbc84) blog post.

# Prerequisites

1. Docker

2. VS Code

# Usage

1. Open this project in the provided [devcontainer](./.devcontainer/devcontainer.json)

2. Run:

    ```bash

    # compile llamacpp and install its dependencies

    make clone-llamacpp-repo

    make compile-llamacpp

    make install-llamacpp-deps

    # get a model and convert it to something llamacpp can use

    make download-model

    make convert-model-to-f16

    make quantize-model

    # view inference timings

    make eval

    ```