https://github.com/dqbd/tiktokenizer

Online playground for OpenAPI tokenizers
https://github.com/dqbd/tiktokenizer

chatgpt nextjs openai t3-stack tiktoken tokenizer

Last synced: about 1 month ago
JSON representation

Online playground for OpenAPI tokenizers

• Host: GitHub
• URL: https://github.com/dqbd/tiktokenizer
• Owner: dqbd
• License: mit
• Created: 2023-03-02T18:24:33.000Z (over 2 years ago)
• Default Branch: master
• Last Pushed: 2025-04-24T08:55:21.000Z (2 months ago)
• Last Synced: 2025-04-24T09:41:05.018Z (2 months ago)
• Topics: chatgpt, nextjs, openai, t3-stack, tiktoken, tokenizer
• Language: TypeScript
• Homepage: https://tiktokenizer.vercel.app
• Size: 709 KB
• Stars: 1,122
• Watchers: 10
• Forks: 131
• Open Issues: 16
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-ChatGPT-repositories - tiktokenizer - Online playground for OpenAPI tokenizers (Openai)
• awesome-chatgpt - dqbd/tiktokenizer - Tiktokenizer is an online playground that accurately calculates the number of tokens in prompts using OpenAI's tokenizers, aiding developers in managing token limits for language models. (UIs / Web applications)
• awesome-chatgpt-api - dqbd/tiktokenizer

README

        
![Tiktokenizer](https://user-images.githubusercontent.com/1443449/222597674-287aefdc-f0e1-491b-9bf9-16431b1b8054.svg)

***

# Tiktokenizer

Online playground for `openai/tiktoken`, calculating the correct number of tokens for a given prompt.

Special thanks to [Diagram](https://diagram.com/) for sponsorship and guidance.

https://user-images.githubusercontent.com/1443449/222598119-0a5a536e-6785-44ad-ba28-e26e04f15163.mp4

## Acknowledgments

- [T3 Stack](https://create.t3.gg/)

- [shadcn/ui](https://github.com/shadcn/ui)

- [openai/tiktoken](https://github.com/openai/tiktoken)