https://github.com/ipfans/genkit-go-llmstxt
a CLI tool to generate llms-full.txt from https://genkit.dev
https://github.com/ipfans/genkit-go-llmstxt
Last synced: 3 days ago
JSON representation
a CLI tool to generate llms-full.txt from https://genkit.dev
- Host: GitHub
- URL: https://github.com/ipfans/genkit-go-llmstxt
- Owner: ipfans
- License: mit
- Created: 2025-07-06T13:48:03.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2025-07-06T14:19:36.000Z (about 1 year ago)
- Last Synced: 2025-10-08T06:47:21.715Z (10 months ago)
- Language: Go
- Size: 54.7 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# genkit-go-llmstxt
A Go tool that scrapes the [Firebase Genkit Go](https://genkit.dev) documentation and converts it to [llms-txt](https://llmstxt.org/).
## Overview
This tool fetches the main Genkit documentation page, extracts all documentation URLs, and converts each page to clean llms-txt. The output is aggregated into a single file that can be used as context for Large Language Models. (~47k tokens)
## Features
- Fetches content from https://genkit.dev
- Automatically discovers documentation URLs using regex pattern matching
- Converts HTML pages to clean llms-txt using readability extraction
- Outputs consolidated documentation to a single file
- Concurrent processing for efficient scraping
## Installation
```bash
go mod download
```
## Usage
### Using Task (recommended)
```bash
task
```
### Using Go directly
```bash
go run main.go > llms-full.txt
```
The tool will:
1. Fetch the main Genkit documentation page
2. Extract all documentation URLs matching the pattern `/go/docs/*`
3. Process each page to extract readable content
4. Output markdown-formatted content to stdout (or redirect to file)
## Dependencies
- [go-readability](https://github.com/mackee/go-readability) - For HTML content extraction and markdown conversion
## License
MIT License - see [LICENSE](LICENSE) file for details.