https://github.com/avilum/llama-saas

A client/server for LLaMA (Large Language Model Meta AI) that can run ANYWHERE.
https://github.com/avilum/llama-saas

ai client-server facebook llama llm nlp

Last synced: 2 months ago
JSON representation

A client/server for LLaMA (Large Language Model Meta AI) that can run ANYWHERE.

• Host: GitHub
• URL: https://github.com/avilum/llama-saas
• Owner: avilum
• License: apache-2.0
• Created: 2023-03-19T13:48:35.000Z (almost 2 years ago)
• Default Branch: master
• Last Pushed: 2023-03-25T20:58:35.000Z (almost 2 years ago)
• Last Synced: 2024-10-18T21:17:33.685Z (2 months ago)
• Topics: ai, client-server, facebook, llama, llm, nlp
• Language: Go
• Homepage:
• Size: 397 KB
• Stars: 60
• Watchers: 4
• Forks: 2
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# llama-saas

A real-time client and server for LLaMA.


- 🚀 Runs on any CPU machine, with no need for GPU 🚀


- The server is written in Go.

- The client is written in Python using requests with response streaming in real time.

I personally used the smallest `7B/` model on an Intel PC / Macbook Pro, which is ~4.8G when quantized to 4 bit, or ~13G in full precision.

# Examples

- Nice example: elaborate about "Github"  



- Biased example: elaborate about "Donald Trump"  



### Get LLaMA Pretrained Checkpoints

Note that LLaMA cannot be used for commercial use.

- > To maintain integrity and prevent misuse, we are releasing our model under a noncommercial license focused on research use cases. Access to the model will be granted on a case-by-case basis to academic researchers; those affiliated with organizations in government, civil society, and academia; and industry research laboratories around the world. People interested in applying for access can find the link to the application in our research paper.

Apply for Official Access. You will get a unique download link once you are approved.

# How to use

Assuming you have the LLaMA checkpoints (☝️)

1. Clone and build https://github.com/ggerganov/llama.cpp

2. Edit the `LLAMA_MODEL_PATH` and `LLAMA_MAIN` variables in `server.go`.

3. Build and run the server: 

```shell

go build

./server

```

4. Run the client:

```shell

python3 -m pip install requests

python3 llama.py

```

## References

1. https://ai.facebook.com/blog/large-language-model-llama-meta-ai/

2. https://github.com/ggerganov/llama.cpp