https://github.com/tensor-fusion/gpt-haskell

A pure Haskell implementation of a decoder-only transformer (GPT)
https://github.com/tensor-fusion/gpt-haskell

deep-learning gpt gpt-2 haskell large-language-models transformer

Last synced: 3 months ago
JSON representation

A pure Haskell implementation of a decoder-only transformer (GPT)

• Host: GitHub
• URL: https://github.com/tensor-fusion/gpt-haskell
• Owner: tensor-fusion
• Created: 2024-05-22T06:07:31.000Z (8 months ago)
• Default Branch: main
• Last Pushed: 2024-06-22T09:59:05.000Z (7 months ago)
• Last Synced: 2024-10-12T23:43:07.119Z (3 months ago)
• Topics: deep-learning, gpt, gpt-2, haskell, large-language-models, transformer
• Language: Haskell
• Homepage:
• Size: 22.5 KB
• Stars: 7
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Haskell GPT

Super minimal implementation of [GPT-2](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) in Haskell.

Highly inspired by:

- [openai/gpt-2](https://github.com/openai/gpt-2).

- [karpathy/minGPT](https://github.com/karpathy/mingpt).

- [picoGPT](https://github.com/jaymody/picoGPT).

Structure is similar to [picoGPT](https://github.com/jaymody/picoGPT). Code contains:

- Translation of OpenAI's BPE Tokenizer

- Decoder-only transformer

- GPT-2 generation

You can run tests with

```

cabal build

cabal test

```

(You'll first need to download GPT-2 weights, tokenizer, and hyperparams into `/models`)

## TODO

- [ ] Merge changes containing top-k sampling and tensorflow inference for actual text generation

- [ ] Nix deps management