https://github.com/jacoblincool/tiktoken-calculator
Calculate the token count for GPT-4, GPT-3.5, GPT-3, and GPT-2.
https://github.com/jacoblincool/tiktoken-calculator
gpt tiktoken tokenizer
Last synced: 4 months ago
JSON representation
Calculate the token count for GPT-4, GPT-3.5, GPT-3, and GPT-2.
- Host: GitHub
- URL: https://github.com/jacoblincool/tiktoken-calculator
- Owner: JacobLinCool
- License: mit
- Created: 2023-03-15T07:14:41.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2023-12-15T08:43:38.000Z (over 2 years ago)
- Last Synced: 2025-05-07T05:47:53.773Z (about 1 year ago)
- Topics: gpt, tiktoken, tokenizer
- Language: Python
- Homepage: https://huggingface.co/spaces/JacobLinCool/tiktoken-calculator
- Size: 68.4 KB
- Stars: 5
- Watchers: 2
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Tiktoken Calculator
Calculate the token count for GPT-4, GPT-3.5, GPT-3, and GPT-2.
> It uses [openai/tiktoken](https://github.com/openai/tiktoken) to calculate the token count.
## How to use
### HuggingFace Spaces
Go to and try it out!
### Docker
There is a pre-built Docker image available on Docker Hub:
```bash
docker run -p 7860:7860 jacoblincool/tiktoken-calculator
```
If you prefer to use Docker Compose, you can clone this repository and run:
```bash
docker compose up -d
```
### Local
I use Poetry to manage dependencies.
Setup the virtual environment after cloning this repository:
```bash
poetry install
```
Then run the app:
```bash
poetry run python app.py
```