{"id":48830769,"url":"https://github.com/ammit/llms-txt","last_synced_at":"2026-04-14T20:30:29.838Z","repository":{"id":349539416,"uuid":"1202773916","full_name":"ammit/llms-txt","owner":"ammit","description":"Generate llms.txt files for any website. Make your site AI-friendly.","archived":false,"fork":false,"pushed_at":"2026-04-06T13:30:18.000Z","size":143,"stargazers_count":1,"open_issues_count":8,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-06T13:30:55.563Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ammit.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-06T11:39:26.000Z","updated_at":"2026-04-06T13:29:00.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ammit/llms-txt","commit_stats":null,"previous_names":["ammit/llms-txt"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/ammit/llms-txt","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ammit%2Fllms-txt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ammit%2Fllms-txt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ammit%2Fllms-txt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ammit%2Fllms-txt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ammit","download_url":"https://codeload.github.com/ammit/llms-txt/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ammit%2Fllms-txt/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31815062,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-14T18:05:02.291Z","status":"ssl_error","status_checked_at":"2026-04-14T18:05:01.765Z","response_time":153,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-14T20:30:29.040Z","updated_at":"2026-04-14T20:30:29.810Z","avatar_url":"https://github.com/ammit.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# llms-txt\n\n[![CI](https://github.com/ammit/llms-txt/actions/workflows/ci.yml/badge.svg)](https://github.com/ammit/llms-txt/actions/workflows/ci.yml)\n[![npm version](https://img.shields.io/npm/v/@ammit/llms-txt.svg)](https://www.npmjs.com/package/@ammit/llms-txt)\n[![npm downloads](https://img.shields.io/npm/dm/@ammit/llms-txt.svg)](https://www.npmjs.com/package/@ammit/llms-txt)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n[![Node.js \u003e=20](https://img.shields.io/node/v/@ammit/llms-txt.svg)](https://nodejs.org)\n\nGenerate `llms.txt` files for any website. Make your site AI-friendly.\n\n[llms.txt](https://llmstxt.org/) is a standard that helps AI agents understand your website. It converts your pages into clean, token-efficient markdown that LLMs can consume directly, reducing token usage by ~90% compared to raw HTML.\n\nOnly 5-15% of websites have `llms.txt` today. This tool generates one for you automatically.\n\n## Install\n\n```bash\nnpm install -g @ammit/llms-txt\n```\n\nOr run directly:\n\n```bash\nnpx @ammit/llms-txt https://example.com\n```\n\n## Usage\n\n```bash\n# Generate llms.txt for a site\nllms-txt https://docs.anthropic.com\n\n# Set crawl depth\nllms-txt https://example.com --depth 3\n\n# Output to files\nllms-txt https://example.com -o ./llms.txt\n\n# Include/exclude URL patterns\nllms-txt https://example.com --include \"/docs/**\" --exclude \"/blog/**\"\n\n# Generate llms-full.txt (all content bundled)\nllms-txt https://example.com --full\n```\n\n## What it does\n\n1. Discovers pages via `sitemap.xml` and link following\n2. Extracts clean content using [Readability](https://github.com/mozilla/readability) (strips nav, ads, scripts)\n3. Converts to markdown via [Turndown](https://github.com/mixmark-io/turndown)\n4. Outputs a standard `llms.txt` index and optional `llms-full.txt` bundle\n\n## Output format\n\nThe generated `llms.txt` follows the [llms.txt standard](https://llmstxt.org/):\n\n```markdown\n# Example Docs\n\n\u003e Documentation for the Example platform.\n\n## Getting Started\n\n- [Quick Start](https://example.com/docs/quickstart): Set up your first project in 5 minutes\n- [Installation](https://example.com/docs/install): System requirements and install steps\n\n## API Reference\n\n- [Authentication](https://example.com/docs/api/auth): API keys and OAuth setup\n- [Endpoints](https://example.com/docs/api/endpoints): Complete REST API reference\n```\n\n## Options\n\n| Flag | Description | Default |\n|------|-------------|---------|\n| `--depth, -d` | Max crawl depth | `3` |\n| `--output, -o` | Output file path | stdout |\n| `--full` | Also generate `llms-full.txt` | `false` |\n| `--include` | URL patterns to include (glob) | all |\n| `--exclude` | URL patterns to exclude (glob) | none |\n| `--rate` | Requests per second | `2` |\n| `--concurrency, -c` | Parallel requests | `5` |\n| `--json` | Output as JSON | `false` |\n| `--verbose, -v` | Show detailed skip/fetch logging | `false` |\n| `--quiet, -q` | Suppress spinner and summary, only raw output | `false` |\n| `--timeout` | Fetch timeout in milliseconds | `10000` |\n| `--lang` | Language filter (e.g. `en`, `de`). Auto-detects from homepage if not set | auto |\n| `--max-pages` | Max pages to crawl (0 = unlimited) | `0` |\n\n## How it works\n\n```\nURL --\u003e Sitemap/Link Discovery --\u003e Content Extraction --\u003e Markdown Conversion --\u003e llms.txt\n```\n\n- **Discovery**: Checks `sitemap.xml` first, falls back to recursive link following\n- **Extraction**: Mozilla Readability isolates main content, removes chrome\n- **Conversion**: Turndown produces clean GFM markdown\n- **Assembly**: Groups pages by URL path into sections, generates descriptions\n\n## Shell Completions\n\n```bash\n# Bash\necho 'eval \"$(llms-txt completion bash)\"' \u003e\u003e ~/.bashrc\n\n# Zsh\necho 'eval \"$(llms-txt completion zsh)\"' \u003e\u003e ~/.zshrc\n\n# Fish\nllms-txt completion fish \u003e ~/.config/fish/completions/llms-txt.fish\n```\n\n## Contributing\n\nContributions welcome. Please open an issue first to discuss what you'd like to change.\n\n## License\n\n[MIT](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fammit%2Fllms-txt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fammit%2Fllms-txt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fammit%2Fllms-txt/lists"}