https://github.com/vinhowe/piston

Train small sequence models in your browser with WebGPU.
https://github.com/vinhowe/piston

deep-learning transformer webgpu

Last synced: 5 months ago
JSON representation

Train small sequence models in your browser with WebGPU.

• Host: GitHub
• URL: https://github.com/vinhowe/piston
• Owner: vinhowe
• License: mit
• Created: 2025-01-16T00:17:15.000Z (over 1 year ago)
• Default Branch: master
• Last Pushed: 2025-11-29T03:01:42.000Z (7 months ago)
• Last Synced: 2025-11-30T11:56:26.150Z (7 months ago)
• Topics: deep-learning, transformer, webgpu
• Language: Rust
• Homepage: https://sequence.toys
• Size: 24.8 MB
• Stars: 15
• Watchers: 1
• Forks: 1
• Open Issues: 4
• Metadata Files:
  • Readme: README.md
  • License: LICENSE
  • Citation: CITATION.cff

Awesome Lists containing this project

README

          


Sequence Toy + Piston







web playground • tweet thread • blog post 



Train language models in your browser with WebGPU-powered autodiff.








Sequence Toy is a web playground for training sequence models (Transformers, LSTMs, GRUs, and vanilla RNNs) using Piston, a proof-of-concept WebGPU automatic differentiation library. This repository houses these related projects.

## Attribution

- Piston is a fork of [Ratchet](https://github.com/huggingface/ratchet), hacked and butchered to add automatic differentiation. I picked Ratchet because it is simple enough to reason about, but it thoughtfully supports WebGPU via [wgpu](https://github.com/gfx-rs/wgpu).

- My implementation of backprop borrows heavily from [Candle](https://github.com/huggingface/candle).

- The lazy execution model is an implementation of [LazyTensor](https://arxiv.org/abs/2102.13267), and borrows from [torch/csrc/lazy](https://github.com/pytorch/pytorch/tree/main/torch/csrc/lazy).

- I used [Keller Jordan](https://x.com/kellerjordan0)'s [Modded-NanoGPT](https://github.com/KellerJordan/modded-nanogpt) as a reference for his Muon optimizer.

- My BPE tokenizer implementation is simplified from [transformers.js](https://github.com/huggingface/transformers.js), which in turn mirrors [transformers](https://github.com/huggingface/transformers).

- My GPT-2 model implementation was originally based on [minGPT](https://github.com/karpathy/minGPT).

- I adapted dataset preprocessing code from [llm.c](https://github.com/karpathy/llm.c).