https://github.com/kspviswa/tiktokenviewer

An attempt to reimplement TikTokenizer
https://github.com/kspviswa/tiktokenviewer

Last synced: 14 days ago
JSON representation

An attempt to reimplement TikTokenizer

• Host: GitHub
• URL: https://github.com/kspviswa/tiktokenviewer
• Owner: kspviswa
• License: mit
• Created: 2024-04-21T10:16:49.000Z (7 months ago)
• Default Branch: main
• Last Pushed: 2024-04-21T15:10:55.000Z (7 months ago)
• Last Synced: 2024-10-11T18:55:00.579Z (about 1 month ago)
• Language: Python
• Size: 3.79 MB
• Stars: 3
• Watchers: 1
• Forks: 2
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# TikTokenViewer

A _childish_ attempt to reimplement [TikTokenizer](https://tiktokenizer.vercel.app/) in pure python using [OpenAI TikToken](https://github.com/openai/tiktoken), [streamlit](https://streamlit.io) and streamlit extras including [st-annotated-text](https://github.com/tvst/st-annotated-text/tree/master) and [st-keyup](https://github.com/blackary/streamlit-keyup)

[Live Demo](https://tiktokenviewer.streamlit.app)

![](tiktokenview_demo.gif)

ToDos:

- [ ] Implement the _hover-over_ effect as seen in original [TikTokenizer](https://tiktokenizer.vercel.app/)

- [ ] Extend / Modify [st-keyup](https://github.com/blackary/streamlit-keyup) to also cover the text area case

- [ ] Show token costs per model