https://github.com/microsoft/Tokenizer

Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.
https://github.com/microsoft/Tokenizer

ai gpt llm openai tokenizer

Last synced: about 1 month ago
JSON representation

Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.

• Host: GitHub
• URL: https://github.com/microsoft/Tokenizer
• Owner: microsoft
• License: mit
• Created: 2023-03-28T07:14:39.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2024-07-18T23:31:28.000Z (about 2 months ago)
• Last Synced: 2024-07-30T00:36:38.395Z (about 1 month ago)
• Topics: ai, gpt, llm, openai, tokenizer
• Language: C#
• Homepage: https://github.com/microsoft/Tokenizer/
• Size: 2.12 MB
• Stars: 129
• Watchers: 9
• Forks: 27
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE
  • Code of conduct: CODE_OF_CONDUCT.md
  • Security: SECURITY.md
  • Support: SUPPORT.md

Awesome Lists containing this project

README

        
# Tokenizer

This repo contains Typescript and C# implementation of byte pair encoding(BPE) tokenizer for OpenAI LLMs, it's based on open sourced rust implementation in the [OpenAI tiktoken](https://github.com/openai/tiktoken). Both implementation are valuable to run prompt tokenization in Nodejs and .NET environment before feeding prompt into a LLM.

## Typescript implementation

Please follow [README](tokenizer_ts/README.md).

## C# implementation

   > [!IMPORTANT]

   > Users of `Microsoft.DeepDev.TokenizerLib` should migrate to `Microsoft.ML.Tokenizers`. The functionality in `Microsoft.DeepDev.TokenizerLib` has been added to [`Microsoft.ML.Tokenizers`](https://www.nuget.org/packages/Microsoft.ML.Tokenizers). `Microsoft.ML.Tokenizers` is a tokenizer library being developed by the .NET team and going forward, the central place for tokenizer development in .NET. By using `Microsoft.ML.Tokenizers`, you should see improved performance over existing tokenizer library implementations, including `Microsoft.DeepDev.TokenizerLib`. A stable release of `Microsoft.ML.Tokenizers` is expected alongside the .NET 9.0 release (November 2024). Instructions for migration can be found at https://github.com/dotnet/machinelearning/blob/main/docs/code/microsoft-ml-tokenizers-migration-guide.md.

## Contributing

We welcome contributions. Please follow [this guideline](CONTRIBUTING.md).

## Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft 

trademarks or logos is subject to and must follow 

[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).

Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.

Any use of third-party trademarks or logos are subject to those third-party's policies.