https://github.com/latitudegames/GPT-3-Encoder

Javascript BPE Encoder Decoder for GPT-2 / GPT-3
https://github.com/latitudegames/GPT-3-Encoder

Last synced: 3 months ago
JSON representation

Javascript BPE Encoder Decoder for GPT-2 / GPT-3

• Host: GitHub
• URL: https://github.com/latitudegames/GPT-3-Encoder
• Owner: latitudegames
• License: mit
• Created: 2020-09-04T18:36:31.000Z (over 4 years ago)
• Default Branch: master
• Last Pushed: 2023-04-16T00:44:43.000Z (almost 2 years ago)
• Last Synced: 2024-11-01T15:41:58.910Z (4 months ago)
• Language: JavaScript
• Size: 651 KB
• Stars: 715
• Watchers: 17
• Forks: 193
• Open Issues: 32
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# GPT-3-Encoder

Javascript BPE Encoder Decoder for GPT-2 / GPT-3

## About

GPT-2 and GPT-3 use byte pair encoding to turn text into a series of integers to feed into the model. This is a javascript implementation of OpenAI's original python encoder/decoder which can be found [here](https://github.com/openai/gpt-2)

## Install with npm

```

npm install gpt-3-encoder

```

## Usage

Compatible with Node >= 12

```js

const {encode, decode} = require('gpt-3-encoder')

const str = 'This is an example sentence to try encoding out on!'

const encoded = encode(str)

console.log('Encoded this string looks like: ', encoded)

console.log('We can look at each token and what it represents')

for(let token of encoded){

  console.log({token, string: decode([token])})

}

const decoded = decode(encoded)

console.log('We can decode it back into:\n', decoded)

```