https://github.com/anasfik/gpt_3_encoder

A pure Dart implementation of OpenAI's original Python encoder/decoder for GPT models.
https://github.com/anasfik/gpt_3_encoder

dart got gpt gpt-2 gpt-3 gpt-4 gpt3 openai openai-api

Last synced: about 1 year ago
JSON representation

A pure Dart implementation of OpenAI's original Python encoder/decoder for GPT models.

• Host: GitHub
• URL: https://github.com/anasfik/gpt_3_encoder
• Owner: anasfik
• License: mit
• Created: 2023-05-07T23:57:40.000Z (about 3 years ago)
• Default Branch: main
• Last Pushed: 2023-05-10T22:17:13.000Z (about 3 years ago)
• Last Synced: 2025-04-03T08:59:55.177Z (about 1 year ago)
• Topics: dart, got, gpt, gpt-2, gpt-3, gpt-4, gpt3, openai, openai-api
• Language: Dart
• Homepage: https://pub.dev/packages/gpt_3_encoder/
• Size: 560 KB
• Stars: 7
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# GPT 3 Encoder

This package aims to provide a simple interface for encoding and decoding text same as GPT-3, GPT-2 that uses byte pair encoding (BPE) to turn text into a series of integers to feed into the models.

This package is a pure Dart implementation of OpenAI's original Python encoder/decoder.

## Usage

```dart

import 'package:gpt_3_encoder/gpt_3_encoder.dart';

void main() {

  // This is the text we want to encode and decode.

  final text = "Hello World!";

  // Encode the text.

  final encoded = GPT3Encoder.instance.encode(text);

  // Print the encoded text and its token length. 

  print(

    "Your text contains ${encoded.length} tokens, encoded as follows: $encoded",

  );

  // Decode back the encoded text token by token and print the results.

  encoded.forEach((token) {

    final decoded = GPT3Encoder.instance.decode([token]);

    print("Token: $token, decoded as: $decoded");

  });

}

```