Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/microsoft/Tokenizer
Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.
https://github.com/microsoft/Tokenizer
ai gpt llm openai tokenizer
Last synced: about 1 month ago
JSON representation
Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.
- Host: GitHub
- URL: https://github.com/microsoft/Tokenizer
- Owner: microsoft
- License: mit
- Created: 2023-03-28T07:14:39.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-07-18T23:31:28.000Z (about 2 months ago)
- Last Synced: 2024-07-30T00:36:38.395Z (about 1 month ago)
- Topics: ai, gpt, llm, openai, tokenizer
- Language: C#
- Homepage: https://github.com/microsoft/Tokenizer/
- Size: 2.12 MB
- Stars: 129
- Watchers: 9
- Forks: 27
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md
- Support: SUPPORT.md
Awesome Lists containing this project
README
# Tokenizer
This repo contains Typescript and C# implementation of byte pair encoding(BPE) tokenizer for OpenAI LLMs, it's based on open sourced rust implementation in the [OpenAI tiktoken](https://github.com/openai/tiktoken). Both implementation are valuable to run prompt tokenization in Nodejs and .NET environment before feeding prompt into a LLM.
## Typescript implementation
Please follow [README](tokenizer_ts/README.md).
## C# implementation
> [!IMPORTANT]
> Users of `Microsoft.DeepDev.TokenizerLib` should migrate to `Microsoft.ML.Tokenizers`. The functionality in `Microsoft.DeepDev.TokenizerLib` has been added to [`Microsoft.ML.Tokenizers`](https://www.nuget.org/packages/Microsoft.ML.Tokenizers). `Microsoft.ML.Tokenizers` is a tokenizer library being developed by the .NET team and going forward, the central place for tokenizer development in .NET. By using `Microsoft.ML.Tokenizers`, you should see improved performance over existing tokenizer library implementations, including `Microsoft.DeepDev.TokenizerLib`. A stable release of `Microsoft.ML.Tokenizers` is expected alongside the .NET 9.0 release (November 2024). Instructions for migration can be found at https://github.com/dotnet/machinelearning/blob/main/docs/code/microsoft-ml-tokenizers-migration-guide.md.## Contributing
We welcome contributions. Please follow [this guideline](CONTRIBUTING.md).
## Trademarks
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
trademarks or logos is subject to and must follow
[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
Any use of third-party trademarks or logos are subject to those third-party's policies.