https://github.com/murabcd/tokenizer
AI model calculator and tokenizer built with Next.js and GPT-3-encoder
https://github.com/murabcd/tokenizer
Last synced: 12 months ago
JSON representation
AI model calculator and tokenizer built with Next.js and GPT-3-encoder
- Host: GitHub
- URL: https://github.com/murabcd/tokenizer
- Owner: murabcd
- Created: 2024-08-27T10:49:01.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2025-03-10T20:50:25.000Z (over 1 year ago)
- Last Synced: 2025-04-20T14:46:29.574Z (about 1 year ago)
- Language: TypeScript
- Homepage: https://oss-tokenizer.vercel.app
- Size: 134 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Tokenizer
This Next.js 14 application serves as an AI model calculator and tokenizer, helping users estimate token counts and costs for various AI models.
## Features
- Select from a range of AI models (cloud-based and local)
- Calculate token count for input text using gpt-3-encoder
- Estimate costs based on selected model and token count
- User-friendly interface with model search and detailed information
- Responsive design using Tailwind CSS
- Lucide React icons for enhanced UI
## Getting Started
### Prerequisites
- Node.js 14.x or later
- npm or Yarn
### Installation
1. Clone the repository:
```
git clone https://github.com/muradpm/tokenizer.git
```
2. Navigate to the project directory:
```
cd tokenizer
```
3. Install dependencies:
```
npm install
# or
yarn install
```
4. Run the development server:
```
npm run dev
# or
yarn dev
```
5. Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.
## Usage
1. Select an AI model from the dropdown or search for a specific model.
2. Enter your text in the input area.
3. View the calculated token count (using gpt-3-encoder) and estimated cost.
4. Hover over model names for additional information.
## Built With
- [Next.js 14](https://nextjs.org/)
- [React](https://reactjs.org/)
- [Tailwind CSS](https://tailwindcss.com/)
- [Lucide React](https://lucide.dev/)
- [GPT-3-encoder](https://www.npmjs.com/package/gpt-3-encoder)
## License
This project is open source and available under the [MIT License](LICENSE).