https://github.com/igorrius/flatten-go-doc
CLI tool and Go library to scrape and flatten pkg.go.dev documentation into a single Markdown file for offline use and LLM context.
https://github.com/igorrius/flatten-go-doc
cli dev-tools documentation go golang markdown notebooklm pkg-go-dev scraper
Last synced: 5 months ago
JSON representation
CLI tool and Go library to scrape and flatten pkg.go.dev documentation into a single Markdown file for offline use and LLM context.
- Host: GitHub
- URL: https://github.com/igorrius/flatten-go-doc
- Owner: igorrius
- License: apache-2.0
- Created: 2025-12-18T11:12:49.000Z (6 months ago)
- Default Branch: main
- Last Pushed: 2025-12-18T15:42:41.000Z (6 months ago)
- Last Synced: 2025-12-21T17:51:23.329Z (6 months ago)
- Topics: cli, dev-tools, documentation, go, golang, markdown, notebooklm, pkg-go-dev, scraper
- Language: Go
- Homepage:
- Size: 47.9 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Flatten Go Doc
`flatten-go-doc` is a CLI tool that scrapes Go package documentation from `pkg.go.dev` and consolidates it into a single, easy-to-read Markdown file. Perfect for offline reading or providing context to LLMs.
## Features
- **GitHub URL Support**: Accepts GitHub repository URLs directly (e.g., `https://github.com/user/repo`) and resolves them to `pkg.go.dev`.
- **Source File Downloading**: Automatically identifies and downloads raw Go source files linked in the documentation, appending them to the output for full context.
- **Retry Logic**: Robust scraping with automatic retries and exponential backoff for transient network issues.
- **Recursive Scraping**: Automatically finds and scrapes sub-packages.
- **Markdown Conversion**: Converts documentation and READMEs into clean Markdown.
- **Sorted Output**: Organizes documentation alphabetically by package URL.
- **SOLID Architecture**: Refactored into modular components (Scraper, Converter, Source Handler) for better maintainability and extensibility.
## Installation
### Using `go install`
You can install `flatten-go-doc` directly to your `$GOPATH/bin`:
```bash
go install github.com/igorrius/flatten-go-doc@latest
```
### From Source
1. Clone the repository:
```bash
git clone https://github.com/igorrius/flatten-go-doc.git
cd flatten-go-doc
```
2. Install locally:
```bash
go install .
```
## Usage
The tool takes the `pkg.go.dev` URL or a GitHub repository URL as the first argument. An optional output file path can be provided as the second argument.
```bash
flatten-go-doc [OUTPUT_FILE]
```
### Examples
**Basic usage (outputs to `.md`):**
```bash
go run . https://pkg.go.dev/github.com/google/go-cmp/cmp
# Creates: github.com_google_go-cmp_cmp.md
```
**Using a GitHub URL:**
```bash
go run . https://github.com/google/go-cmp
# Resolves to pkg.go.dev and creates: github.com_google_go-cmp.md
```
**Specifying an output file:**
```bash
flatten-doc https://pkg.go.dev/github.com/google/go-cmp/cmp my-docs.md
```
## Library Usage
You can also use the scraper as a library in your own Go projects:
```go
import "github.com/igorrius/flatten-go-doc/pkg/flattener"
func main() {
config := flattener.DefaultConfig()
f := flattener.New(config)
results, err := f.Flatten("https://pkg.go.dev/some/package")
// ... handle results
}
```
## Tip: Using with NotebookLM
`flatten-go-doc` is an excellent companion for Google **NotebookLM**. By consolidating documentation and source code into a single Markdown file, you can:
1. **Full Library Context**: Upload the generated `.md` file as a source in NotebookLM.
2. **AI-Powered Guidance**: Ask NotebookLM to "explain how to use this library based on the source code" or "create a tutorial for [specific feature]".
3. **Code Synthesis**: Since the tool includes the actual `.go` source files, NotebookLM can reason about the implementation details that aren't always visible in the standard documentation, helping you write more accurate and idiomatic code.
## License
This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.