Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/fatescript/token_visualizer
Token level visualization tools for large language models
https://github.com/fatescript/token_visualizer
Last synced: 6 days ago
JSON representation
Token level visualization tools for large language models
- Host: GitHub
- URL: https://github.com/fatescript/token_visualizer
- Owner: FateScript
- Created: 2024-03-06T12:51:24.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2024-09-25T06:52:13.000Z (4 months ago)
- Last Synced: 2024-12-26T15:09:02.346Z (13 days ago)
- Language: Python
- Homepage:
- Size: 605 KB
- Stars: 63
- Watchers: 2
- Forks: 7
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## Introduction
token visualizer is a token-level visualization tool to visualize LLM.
## Quick start
### Installation
#### Install from source
Run the following command to install the package.
```shell
git clone [email protected]:FateScript/token_visualizer.git
cd token_visualizer
pip install -v -e . # or python3 setup.py develop
```
#### Check installation
If you could see the version of `token_visualizer` by running
```shell
python3 -c "import token_visualizer; print(token_visualizer.__file__)"
```
## Visualization demo
### Inference
#### Start demo
Run the following command to start inference visualizer.
```shell
python3 visualizer.py
```
The command will start a OpenAIProxy model, to use it without exception, user should fill in the value of `BASE_URL` and `OPENAI_KEY`.
`token_visualizer` also support `OpenAIModel` and HuggingFace `TransformerModel` in [models.py](https://github.com/FateScript/token_visualizer/blob/main/token_visualizer/models.py), feel free to modify the code.
#### Demo gif
After inputing your prompt, you will see the large language model's answer and the answer's visualization result.
**The redder the color of the token, the lower the corresponding probability. The greener the color of the token, the higher the corresponding probability.**
### Perplexity
#### Start demo
Run the following command to start perplexity visualizer, then click the `ppl` tab.
```shell
python3 visualizer.py
```
#### Demo gif
After inputing your text, you will see the perplexity and visualization result of the text.
### Tokenizer
#### Start demo
Run the following command to start interactive tokenizer encoding web demo.
```shell
python3 visual_tokenizer.py
```
#### Demo gif
User could select tokenizer to interacte with and text to encode. For speical string
## TODO
- [x] Support ppl visualization.
- [x] Select transformers/openai/TGI with cli.
- [ ] Support OpenAI tokenizer visualization.
- [x] Support TGI inference visualization.
- [ ] Support multi-turn chat visualization.
- [ ] Support dark mode.
## Related projects/websites
* [LLM architecture visualization](https://bbycroft.net/llm)
* [perplexity visualization](https://bbycroft.net/ppl)
## Acknowledgement
* Use front-end setting from [https://perplexity.vercel.app/](https://bbycroft.net/ppl)
* Color algorithm from [post](https://twitter.com/thesephist/status/1617909119423500288) by [thesephist](https://twitter.com/thesephist).