https://github.com/ayub-kokabi/llmtext
Turns any website into a single LLM-ready markdown file
https://github.com/ayub-kokabi/llmtext
ai cli devtools llm rust
Last synced: 4 months ago
JSON representation
Turns any website into a single LLM-ready markdown file
- Host: GitHub
- URL: https://github.com/ayub-kokabi/llmtext
- Owner: ayub-kokabi
- License: mit
- Created: 2025-07-27T14:40:16.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2025-08-03T16:31:55.000Z (11 months ago)
- Last Synced: 2026-01-14T01:33:55.087Z (5 months ago)
- Topics: ai, cli, devtools, llm, rust
- Language: Rust
- Homepage:
- Size: 51.8 KB
- Stars: 13
- Watchers: 0
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: Readme.md
Awesome Lists containing this project
README
# llmtext 🤖🔗📝
Turn any website into a single, clean Markdown file, ready for your LLM prompts.
Language models have a knowledge cut-off date. `llmtext` solves this by scraping a webpage and all its relevant internal links, converting the content into a single Markdown file. You can then use this file to give your LLM the up-to-date context it needs.
[](https://crates.io/crates/llmtext)
[](LICENSE)
---
https://github.com/user-attachments/assets/a558970e-4d75-4b41-8c2c-3bb3d5705e13
## ✨ Features
- **Intelligent Scraping**: Give it one URL, and it smartly finds and scrapes all related pages.
- **Flexible Input**: Scrape a single page (`--single`) or a list of URLs from a file (`--urls`).
- **Fast & Efficient**: Built in Rust for maximum speed with parallel downloads.
- **Clipboard Integration**: Instantly copy the full Markdown output to your clipboard (`--clipboard`).
## 📦 Installation
```bash
# From Crates.io (Recommended)
cargo install llmtext
```
## 🚀 Usage
```bash
# Scrape a site's documentation section
llmtext https://react.dev/reference/react
# Scrape just a single page
llmtext --single https://react.dev/reference/react/useState
# Scrape a list of URLs from a file and save to a custom output file
llmtext --urls my_links.txt --output react-subset.md
# Scrape and copy directly to clipboard, skipping confirmation
llmtext https://vuejs.org/guide/introduction.html --clipboard --yes
```
## License
This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.