{"id":31620798,"url":"https://github.com/derailed-dash/llms-generator","last_synced_at":"2026-04-16T00:32:03.858Z","repository":{"id":315967981,"uuid":"1060315380","full_name":"derailed-dash/LLMs-Generator","owner":"derailed-dash","description":"An agentic solution designed to create a llms.txt file for any given repo or folder.","archived":false,"fork":false,"pushed_at":"2025-09-21T21:46:03.000Z","size":373,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-21T23:33:36.442Z","etag":null,"topics":["adk","adk-python","agentic-ai","gemini","multi-agent-systems","python"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/derailed-dash.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-19T17:53:40.000Z","updated_at":"2025-09-21T21:46:06.000Z","dependencies_parsed_at":"2025-09-21T23:33:38.611Z","dependency_job_id":"2f3f18df-52d3-40f2-9e8a-570fd372fa92","html_url":"https://github.com/derailed-dash/LLMs-Generator","commit_stats":null,"previous_names":["derailed-dash/llms-generator"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/derailed-dash/LLMs-Generator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derailed-dash%2FLLMs-Generator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derailed-dash%2FLLMs-Generator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derailed-dash%2FLLMs-Generator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derailed-dash%2FLLMs-Generator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/derailed-dash","download_url":"https://codeload.github.com/derailed-dash/LLMs-Generator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derailed-dash%2FLLMs-Generator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278638064,"owners_count":26019939,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-06T02:00:05.630Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adk","adk-python","agentic-ai","gemini","multi-agent-systems","python"],"created_at":"2025-10-06T15:42:48.989Z","updated_at":"2026-04-16T00:32:03.838Z","avatar_url":"https://github.com/derailed-dash.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LLMS-Generator\n\n## Table of Contents\n\n- [Repo Metadata](#repo-metadata)\n- [Repo Overview](#repo-overview)\n- [Getting Started](#getting-started)\n  - [Prerequisites](#prerequisites)\n  - [Environment Variables](#environment-variables)\n  - [Installation](#installation)\n- [How to Use](#how-to-use)\n  - [Command](#command)\n  - [Arguments](#arguments)\n  - [Options](#options)\n\n- [How to Use the Generated llms.txt](#how-to-use-the-generated-llmstxt)\n- [Associated Articles](#associated-articles)\n- [Useful Commands](#useful-commands)\n  - [Testing](#testing)\n  - [Running in a Local Container](#running-in-a-local-container)\n\n## Repo Metadata\n\nAuthor: Darren Lester\n\n## Repo Overview\n\n_LLMS-Generator_ is an agentic solution designed to create a `llms.txt` file for any given repo or folder. It intelligently processes large codebases by batching files and iteratively summarizing them to overcome model context limitations.\n\nThe `llms.txt` file is an AI/LLM-friendly markdown file that enables an AI to understand the purpose of the a repo, as well as have a full understanding of the repo site map and the purpose of each file it finds. This is particularly useful when providing AIs (like Gemini) access to documentation repos.\n\nAn `llms.txt` file should have this structure:\n\n- An `H1` with the name of the project or site\n- An overview of the project / site purpose.\n- Zero or more markdown sections delimited by `H2` headers, containing appropriate section summaries.\n- Each section contains a list of of markdown hyperlinks, in the format: `[name](url): summary`.\n\nSee [here](https://github.com/AnswerDotAI/llms-txt) for a more detailed description of the `llms.txt` standard.\n\n## How to Use the Generated llms.txt\n\nAn AI can easily read the `llms.txt` and follow the links it finds there. When you ask your agent a deep-dive question about a topic, the agent will be able to follow the appropriate links to give you grounded answers.\n\nIt is easy to provide an agent with the ability to consume an `llms.txt` file with an MCP server, like this one: [ADK-Docs-Ext](https://github.com/derailed-dash/adk-docs-ext).\n\n## Related Links and Docs\n\n- [Give Your AI Agents Deep Understanding With LLMS.txt](https://medium.com/google-cloud/give-your-ai-agents-deep-understanding-with-llms-txt-4f948590332b)\n- [Give Your AI Agents Deep Understanding - Creating the LLMS.txt with a Multi-Agent ADK Solution - Coming Soon](tbd)\n- [ADK Docs Extension for Gemini CLI](https://github.com/derailed-dash/adk-docs-ext)\n- [Agent Patterns](https://cloud.google.com/architecture/choose-design-pattern-agentic-ai-system)\n\n## Solution Design\n\n![Solution Design Diagram](docs/generate-llms-adk.drawio.png)\n\nCheck the design [here](docs/solution-design.md).\n\n## Getting Started\n\nTo get started with LLMS-Generator, follow these steps:\n\n### Prerequisites\n\n*   **uv:** Ensure you have `uv` installed for Python package and environment management. If not, you can install it by following the instructions on the [uv website](https://astral.sh/uv/install/).\n*   **Google Cloud SDK:** Install the Google Cloud SDK to interact with GCP services. Follow the official [Google Cloud SDK documentation](https://cloud.google.com/sdk/docs/install) for installation instructions.\n*   **make:** Ensure `make` is installed on your system. It's typically available on most Unix-like systems.\n\n### Environment Variables\n\nThis project uses a `.env` file to manage environment variables. Before running the application, you need to create a `.env` file in the root of the project.\n\nYou can copy the example below and customize it with your own values.\n\n```bash\n# .env\n\nexport GOOGLE_CLOUD_STAGING_PROJECT=\"your-staging-project-id\"\nexport GOOGLE_CLOUD_PRD_PROJECT=\"your-prod-project-id\"\n\n# These Google Cloud variables are set by the scripts/setup-env.sh script\n# GOOGLE_CLOUD_PROJECT=\"\"\n# GOOGLE_CLOUD_LOCATION=\"global\"\n\nexport PYTHONPATH=\"src\"\n\n# Agent variables\nexport AGENT_NAME=\"llms_gen_agent\" # The name of the agent\nexport MODEL=\"gemini-2.5-flash\" # The model used by the agent\nexport GOOGLE_GENAI_USE_VERTEXAI=\"True\" # True to use Vertex AI for auth; else use API key\nexport LOG_LEVEL=\"INFO\"\nexport MAX_FILES_TO_PROCESS=10 # Set to 0 for no limit\n\n# Exponential backoff parameters for the model API calls\nexport BACKOFF_INIT_DELAY=5\nBACKOFF_ATTEMPTS=5\nBACKOFF_MAX_DELAY=60\nBACKOFF_MULTIPLIER=2 # exponential delay growth\n```\n\n### Installation\n\n1.  **Clone the repository:**\n    ```bash\n    git clone https://github.com/derailed-dash/llms-generator.git\n    cd llms-generator\n    ```\n2.  **Set up your environment:**\n    Run the setup script to configure your Google Cloud project and authentication, and load environment variables from `.env`.\n    ```bash\n    source scripts/setup-env.sh\n    ```\n    This script will guide you through setting up the necessary environment variables and authenticating with Google Cloud.\n3.  **Install dependencies:**\n    Use `make install` to install all required Python dependencies using `uv`.\n    ```bash\n    make install\n    ```\n\nAfter completing these steps, your environment will be set up, and all dependencies will be installed, ready for development or running the agent.\n\n## How to Use\n\nOnce the dependencies are installed and the environment is set up, you can use the `llms-gen` command-line tool to generate the `llms.txt` file.\n\n### Command\n\nThe `llms-gen` command-line application is exposed via the `[project.scripts]` section in `pyproject.toml`. When the package is installed, this entry point allows you to run `llms-gen` directly from your terminal, which executes the `app` object defined in `src/client_fe/cli.py`.\n\n```bash\nllms-gen --repo-path /path/to/your/repo [OPTIONS]\n```\n\nE.g.\n\n```bash\nllms-gen --repo-path \"/home/darren/localdev/gcp/adk-docs\"\n```\n\n### Arguments\n\n*   `--repo-path` / `-r`: (Required) The absolute path to the repository to generate the `llms.txt` file for.\n\n### Options\n\n*   `--output-path` / `-o`: The absolute path to save the `llms.txt` file. If not specified, it will be saved in a `temp` directory in the current working directory.\n*   `--log-level` / `-l`: Set the log level for the application (e.g., `DEBUG`, `INFO`, `WARNING`, `ERROR`). This will override any `LOG_LEVEL` environment variable.\n\n## Useful Commands\n\n| Command                       | Description                                                                           |\n| ----------------------------- | ------------------------------------------------------------------------------------- |\n| `source scripts/setup-env.sh` | Setup Google Cloud project and auth with Dev/Staging. Parameter options:\u003cbr\u003e `[--noauth] [-t\\|--target-env \u003cDEV\\|PROD\u003e]` |\n| `make install`                | Install all required dependencies using `uv` |\n| `make playground`             | Launch UI for testing agent locally and remotely. This runs `uv run adk web src` |\n| `make test`                   | Run unit and integration tests |\n| `make lint`                   | Run code quality checks (codespell, ruff, mypy) |\n| `make generate`               | Execute the Llms-Generator command line application |\n| `uv run jupyter lab`          | Launch Jupyter notebook |\n\nFor full command options and usage, refer to the [Makefile](Makefile).\n\n### Testing\n\n- All tests are in the `src/tests` folder.\n- We can run our tests with `make test`.\n- Note that integration tests will fail if the development environment has not first been configured with the `setup-env.sh` script. This is because the test code will not have access to the required Google APIs.\n- If we want to run tests verbosely, we can do this:\n\n  ```bash\n  uv run pytest -v -s src/tests/unit/test_name.py\n  ```\n\n#### Testing Locally\n\nWith ADK CLI:\n\n```bash\nuv run adk run src/llms_gen_agent\n```\n\nWith GUI:\n\n```bash\n# Last param is the location of the agents\nuv run adk web src\n\n# Or we can use the Agent Starter Git make aliases\nmake install \u0026\u0026 make playground\n```\n\n### Running in a Local Container\n\n```bash\n# from project root directory\n\n# Get a unique version to tag our image\nexport VERSION=$(git rev-parse --short HEAD)\n\n# To build as a container image\ndocker build -t $SERVICE_NAME:$VERSION .\n\n# To run as a local container\n# We need to pass environment variables to the container\n# and the Google Application Default Credentials (ADC)\ndocker run --rm -p 8080:8080 \\\n  -e GOOGLE_CLOUD_PROJECT=$GOOGLE_CLOUD_PROJECT -e GOOGLE_CLOUD_REGION=$GOOGLE_CLOUD_REGION \\\n  -e LOG_LEVEL=$LOG_LEVEL \\\n  -e APP_NAME=$APP_NAME \\\n  -e AGENT_NAME=$AGENT_NAME \\\n  -e GOOGLE_GENAI_USE_VERTEXAI=$GOOGLE_GENAI_USE_VERTEXAI \\\n  -e MODEL=$MODEL \\\n  -e GOOGLE_APPLICATION_CREDENTIALS=\"/app/.config/gcloud/application_default_credentials.json\" \\\n  --mount type=bind,source=${HOME}/.config/gcloud,target=/app/.config/gcloud \\\n   $SERVICE_NAME:$VERSION\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderailed-dash%2Fllms-generator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fderailed-dash%2Fllms-generator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderailed-dash%2Fllms-generator/lists"}