Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/ssoudan/nanogptrs
Rust implementation of a simple language model.
https://github.com/ssoudan/nanogptrs
language-model rust
Last synced: 6 days ago
JSON representation
Rust implementation of a simple language model.
- Host: GitHub
- URL: https://github.com/ssoudan/nanogptrs
- Owner: ssoudan
- License: apache-2.0
- Created: 2023-03-28T18:13:25.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-12-13T22:30:26.000Z (25 days ago)
- Last Synced: 2024-12-13T23:25:00.824Z (25 days ago)
- Topics: language-model, rust
- Language: Rust
- Homepage:
- Size: 966 KB
- Stars: 15
- Watchers: 3
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE-APACHE
Awesome Lists containing this project
README
# nanoGPT(rs)
This is a Rust implementation of the nanoGPT model from Andrej Karpathy's YT
video: https://www.youtube.com/watch?v=kCc8FmEb1nY&t=12s
With some help from: https://github.com/LaurentMazare/tch-rs/blob/main/examples/min-gpt/main.rs and
https://github.com/karpathy/nanoGPT/blob/master/model.py.
# Setup
Create micromamba environment (or conda):
```bash
micromamba env create -f environment.yml
```
Activate environment:
```bash
micromamba activate nanogptrs
```
```bash
export LD_LIBRARY_PATH=/opt/conda/lib/python3.10/site-packages/torch/lib/:$LD_LIBRARY_PATH
```
# Run - generate from pretrained GPT-2
```bash
./data/download.sh
./models/download.sh gpt2
cargo run --release -- --device=cuda --restore-from models/gpt2/model.safetensors generate --max-len 32 --prompt "Once upon a time" gpt2
```
# Run - train nano-gpt
```bash
cargo run --release -- --device=cuda train --n-epochs=3 --final-checkpoint-path=models/nanogptrs.safetensors nano-gpt
```
Should eventually (~5h on my Titan XP) produce something like this:
```
DUCHESS OF YORK:
Here comes already.
EXTOLY:
O, by the means of your crown?
KING HENRY VI:
Brother, that my lord, change thou givest queen.
KING RICHARD II:
Mine honour, because I am advertised
The queen our is not your voice. Would thy sight
Next Rome, among, insible express to dictliffe:
For ere for goings
Abova drunking redel her food pain soul to every it.
QUEEN MARGARET:
I took! O, if you so, good and the Montague of slave,
That he's breathing which holy a holy brats.
```