https://github.com/zemlyansky/gpt-tfjs

GPT in TensorFlow.js
https://github.com/zemlyansky/gpt-tfjs

gpt tensorflowjs tfjs transformers

Last synced: 15 days ago
JSON representation

GPT in TensorFlow.js

• Host: GitHub
• URL: https://github.com/zemlyansky/gpt-tfjs
• Owner: zemlyansky
• License: mit
• Created: 2023-04-23T06:05:35.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2023-10-16T20:10:53.000Z (9 months ago)
• Last Synced: 2024-06-17T21:42:56.156Z (19 days ago)
• Topics: gpt, tensorflowjs, tfjs, transformers
• Language: JavaScript
• Homepage: https://www.npmjs.com/package/gpt-tfjs
• Size: 40 KB
• Stars: 24
• Watchers: 6
• Forks: 5
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • License: LICENSE.md

Lists

• awesome-stars - zemlyansky/gpt-tfjs - GPT in TensorFlow.js (others)

README

        
# GPT in TensorFlow.js (gpt-tfjs)

A minimal implementation of GPT architecture in TensorFlow.js more or less following the [minGPT](https://github.com/karpathy/minGPT) implementation in PyTorch. 

### Installation

```bash

npm i gpt-tfjs

```

### Example 

```javascript

const tf = require('@tensorflow/tfjs') // or @tensorflow/tfjs-node or @tensorflow/tfjs-node-gpu

const { GPTLMHeadModel } = require('gpt-tfjs').model

const config = {

  nLayer: 3,

  nHead: 3,

  nEmbd: 48,

  vocabSize: 3,

  blockSize: 11,

  dropout: 0.1

}

(async () => {

  const gpt = GPTLMHeadModel(config)

  await gpt.train(trainDataset, {epochs: 10, verbose: true})

  const inputs = [2, 2, 2, 1, 0, 1]

  const idx = await gpt.generate([inputs], { maxNewTokens: 6 }) // Or gpt.generateSync(..., {...})

  console.log(idx.arraySync()[0].slice(6)) // [0, 1, 1, 2, 2, 2]

})()

```

Where `trainDataset` is a tensorflow dataset, for example: [https://github.com/zemlyansky/gpt-tfjs/blob/main/projects/sort/sort.js](https://github.com/zemlyansky/gpt-tfjs/blob/main/projects/sort/sort.js)

### Testing

The testing script relies on minGPT to generate the test data. Before running tests run the following command (you'll need Pytorch installed):

```bash

python test_gen.py

```

### Demo

- [https://jsee.io/gpt2-tfjs/](https://jsee.io/gpt2-tfjs/) - Loading GPT-2 weights from Hugginface and running inference in the browser