https://github.com/dqbd/tiktokenizer

Online playground for OpenAPI tokenizers
https://github.com/dqbd/tiktokenizer

chatgpt nextjs openai t3-stack tiktoken tokenizer

Last synced: 1 day ago
JSON representation

Online playground for OpenAPI tokenizers

• Host: GitHub
• URL: https://github.com/dqbd/tiktokenizer
• Owner: dqbd
• License: mit
• Created: 2023-03-02T18:24:33.000Z (almost 2 years ago)
• Default Branch: master
• Last Pushed: 2024-10-17T20:42:15.000Z (3 months ago)
• Last Synced: 2024-10-29T17:38:30.658Z (3 months ago)
• Topics: chatgpt, nextjs, openai, t3-stack, tiktoken, tokenizer
• Language: TypeScript
• Homepage: https://tiktokenizer.vercel.app
• Size: 706 KB
• Stars: 700
• Watchers: 6
• Forks: 87
• Open Issues: 15
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-ChatGPT-repositories - tiktokenizer - Online playground for OpenAPI tokenizers (Openai)
• awesome-chatgpt - dqbd/tiktokenizer - Online playground for OpenAPI tokenizers, allowing users to calculate the correct number of tokens for a given prompt (Other / Other sdk/libraries)
• awesome-chatgpt-api - dqbd/tiktokenizer

README

        
![Tiktokenizer](https://user-images.githubusercontent.com/1443449/222597674-287aefdc-f0e1-491b-9bf9-16431b1b8054.svg)

***

# Tiktokenizer

Online playground for `openai/tiktoken`, calculating the correct number of tokens for a given prompt.

Special thanks to [Diagram](https://diagram.com/) for sponsorship and guidance.

https://user-images.githubusercontent.com/1443449/222598119-0a5a536e-6785-44ad-ba28-e26e04f15163.mp4

## Acknowledgments

- [T3 Stack](https://create.t3.gg/)

- [shadcn/ui](https://github.com/shadcn/ui)

- [openai/tiktoken](https://github.com/openai/tiktoken)