https://github.com/karn/llama-droid
PoC for running alpaca.cpp for on-device, offline, inference on Android
https://github.com/karn/llama-droid
Last synced: 12 months ago
JSON representation
PoC for running alpaca.cpp for on-device, offline, inference on Android
- Host: GitHub
- URL: https://github.com/karn/llama-droid
- Owner: Karn
- Created: 2023-12-28T07:36:54.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-12-28T08:02:29.000Z (over 2 years ago)
- Last Synced: 2025-05-31T13:02:45.804Z (about 1 year ago)
- Language: C
- Homepage:
- Size: 276 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# LLama-Droid
PoC for running [Alpaca](https://github.com/rupeshs/alpaca.cpp) on-device. That uses a modified version of alpaca.cpp to perform inference.
### Requirements
- A device with a >8GB storage, and enough RAM to load the model into memory (>6GB). Download [ggml-alpaca-7b-q4.bin](https://github.com/Karn/alpaca.cpp/tree/master?tab=readme-ov-file#get-started), rename it to `alpaca7b.bin` and upload it to the device in the Application Files Directory under `models`.
- Time, since on-device inference is currently incredibly slow.
### Future work
- What does GPU accelerated inference look like?
- mmap? How, and does it make a difference.
- Consider using a smaller model to improve speed.