https://github.com/openai/gpt-3-encoder
Javascript BPE Encoder Decoder for GPT-2 / GPT-3
https://github.com/openai/gpt-3-encoder
Last synced: 3 months ago
JSON representation
Javascript BPE Encoder Decoder for GPT-2 / GPT-3
- Host: GitHub
- URL: https://github.com/openai/gpt-3-encoder
- Owner: openai
- License: mit
- Fork: true (latitudegames/GPT-3-Encoder)
- Created: 2020-11-23T17:05:55.000Z (about 5 years ago)
- Default Branch: master
- Last Pushed: 2023-04-02T11:52:45.000Z (over 2 years ago)
- Last Synced: 2025-01-19T02:32:50.147Z (11 months ago)
- Size: 614 KB
- Stars: 122
- Watchers: 12
- Forks: 92
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
GPT-3-Encoder
Javascript BPE Encoder Decoder for GPT-2 / GPT-3
## About
GPT-2 and GPT-3 use byte pair encoding to turn text into a series of integers to feed into the model. This is a javascript implementation of OpenAI's original python encoder/decoder which can be found [here](https://github.com/openai/gpt-2)
## Install with npm
`npm install gpt-3-encoder`
## Usage
Compatible with Node >= 12
```
const {encode, decode} = require('gpt-3-encoder')
const str = 'This is an example sentence to try encoding out on!'
const encoded = encode(str)
console.log('Encoded this string looks like: ', encoded)
console.log('We can look at each token and what it represents')
for(let token of encoded){
console.log({token, string: decode([token])})
}
const decoded = decode(encoded)
console.log('We can decode it back into:\n', decoded)
```