https://github.com/yuma-shintani/chunksize-checker
Calculate the number of total tokens, optimal chunk size and chunk overlap from any given document.
https://github.com/yuma-shintani/chunksize-checker
chunking electron rag
Last synced: 2 months ago
JSON representation
Calculate the number of total tokens, optimal chunk size and chunk overlap from any given document.
- Host: GitHub
- URL: https://github.com/yuma-shintani/chunksize-checker
- Owner: yuma-shintani
- License: mit
- Created: 2025-02-05T08:44:58.000Z (4 months ago)
- Default Branch: main
- Last Pushed: 2025-02-14T03:04:28.000Z (4 months ago)
- Last Synced: 2025-02-14T04:19:30.809Z (4 months ago)
- Topics: chunking, electron, rag
- Language: JavaScript
- Homepage:
- Size: 1.14 MB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Chunk Size Checker
Calculate the number of total tokens, optimal chunk size and chunk overlap from any given document.
## Features
- Calculate the number of total tokens, optimal chunk size and chunk overlap.
- Support multi document
- Support xlsx, pptx, docx, pdf, csv, md, txt, log, json
## Installation
### For End Users
1. Download the ChunkSize Checker latest.
2. Use it just like a normal app.
### For Developers
1. Fork the repository on GitHub.
2. Clone your forked repository to your local machine.
3. Install dependencies and run the app.
```
git clone https://github.com/yuma-shintani/chunksize-checker.git
cd chunksize-checker
npm run install
npm run dev
```
## License
This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for more details.
## Contact
For any inquiries or support, please open an issue on GitHub.