{"id":22937638,"url":"https://github.com/tanq16/ai-context","last_synced_at":"2025-09-11T10:39:31.796Z","repository":{"id":267897417,"uuid":"902685829","full_name":"Tanq16/ai-context","owner":"Tanq16","description":"CLI tool to produce MD context files from many sources, to help interact with LLMs (ChatGPT, Llama3, Claude, etc.).","archived":false,"fork":false,"pushed_at":"2025-03-19T15:44:57.000Z","size":6975,"stargazers_count":83,"open_issues_count":0,"forks_count":7,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-02T16:07:31.593Z","etag":null,"topics":["ai","anthropic","chatgpt","claude","cli","context","data-hoarder","deepseek","gemini","genai","github","golang","gpt","llama3","llm","llm-context","ollama","webpage-capture","youtube"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Tanq16.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-13T04:09:25.000Z","updated_at":"2025-03-27T06:48:17.000Z","dependencies_parsed_at":"2025-02-02T08:18:19.515Z","dependency_job_id":"09beecc5-b841-4ce8-86a4-954e8aa0f350","html_url":"https://github.com/Tanq16/ai-context","commit_stats":null,"previous_names":["tanq16/ai-context"],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tanq16%2Fai-context","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tanq16%2Fai-context/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tanq16%2Fai-context/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tanq16%2Fai-context/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Tanq16","download_url":"https://codeload.github.com/Tanq16/ai-context/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248074927,"owners_count":21043491,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","anthropic","chatgpt","claude","cli","context","data-hoarder","deepseek","gemini","genai","github","golang","gpt","llama3","llm","llm-context","ollama","webpage-capture","youtube"],"created_at":"2024-12-14T12:13:51.964Z","updated_at":"2025-09-11T10:39:31.786Z","avatar_url":"https://github.com/Tanq16.png","language":"Go","readme":"\u003cdiv align=\"center\"\u003e\n\n\u003cimg src=\".github/assets/logo.png\" alt=\"AI Context Logo\" width=\"250\"/\u003e\n\n\u003ch1\u003eAI Context\u003c/h1\u003e\n\n[![Release Build](https://github.com/tanq16/ai-context/actions/workflows/build-release.yml/badge.svg)](https://github.com/tanq16/ai-context/actions/workflows/build-release.yml)\n[![GitHub Release](https://img.shields.io/github/v/release/tanq16/ai-context)](https://github.com/Tanq16/ai-context/releases/latest)\n\nGenerate AI-friendly markdown files from GitHub repos, local code, YouTube videos, or webpages using a multi-arch, multi-OS CLI tool to make your interactions with LLMs (like ChatGPT, Claude, etc.) easy. AI Context can also serve a web frontend for self-hosting.\n\n\u003ca href=\"#features\"\u003eFeatures\u003c/a\u003e \u0026bull; \u003ca href=\"#installation\"\u003eInstallation\u003c/a\u003e \u0026bull; \u003ca href=\"#usage\"\u003eUsage\u003c/a\u003e \u0026bull; \u003ca href=\"#acknowledgments\"\u003eAcknowledgements\u003c/a\u003e\n\n\u003c/div\u003e\n\n---\n\n`Quickstart` \u0026rarr;\n\n```bash\nai-context \"https://github.com/tanq16/ai-context\" # single URL\nai-context -f urllist.file                        # URL file\nai-context serve                                  # serve with web frontend\n```\n\nFor self-hosting the web frontend as a Docker container, use:\n\n```bash\ndocker run --rm --name \"ai-context\" -d tanq16/ai-context:main\n```\n\n## Features\n\n- **Local Directory Processing**\n    - this is mainly for locally available code bases (directories or already cloned git repos)\n    - the context file includes directory structure and all file contents within context\n- **GitHub Repository Processing**\n    - this clones and processes provided GitHub link and does the same as *Local Directory Processing*\n    - it temporarily clones the repository, so no need for cleanup\n    - it also supports private repositories on GitHub through use of `GH_TOKEN` environment variable\n- **YouTube Transcript Processing**\n    - this downloads transcripts for given YouTube video link and stores it as markdown\n    - the transcript also preserves time segments\n- **WebPage Processing**\n    - this converts an HTML webpage to markdown text, stripping off JS and CSS\n    - it also downloads all images from the page and stores them locally with UUID filenames\n    - the markdown text includes links via local paths to the downloaded images\n- **Self-Hosted Application Variant**\n    - you can self-host the application for quick access from devices with browsers to get context\n    - web frontend uses a simple but elegant Tailwind-UI to allow copy/download of context from all sources\n\n## Installation\n\n- **Binary**\n    - Download the latest release for your platform and OS from the [releases page](https://github.com/tanq16/ai-context/releases)\n    - Binaries are build via GitHub actions for MacOS, Linux, and Windows for both AMD64 (x86_64) and ARM64 (like Apple Silicon) architectures\n    - You can also download specific versions if needed; however, the latest version is recommended\n- **Go Install**\n    - Run the following command (requires `Go v1.22+`):\n    ```bash\n    go install github.com/tanq16/ai-context@latest\n    ```\n    - For specific versions, use binaries or build specific commits as I have not and will not implement Go-native binary versioning\n- **Local Build**\n    ```bash\n    git clone https://github.com/tanq16/ai-context.git \u0026\u0026 \\\n    cd ai-context\n    ```\n    ```bash\n    go build .\n    ```\n\n## Usage\n\n### Screencasts\n\n| | |\n| --- | --- |\n| UI | \u003cimg src=\".github/assets/ui1.gif\" width=\"62%\"\u003e \u003cimg src=\".github/assets/ui2.gif\" width=\"22%\"\u003e |\n| CLI | \u003cimg src=\".github/assets/cli1.gif\" width=\"42%\"\u003e \u003cimg src=\".github/assets/cli2.gif\" width=\"42%\"\u003e |\n\n### Primary Usage\n\n```bash\n# Process a single path (local directory) with additional ignore patterns\nai-context /path/to/directory  -i \"tests,docs,*doc.*\"\n\n# Process one URL (GitHub repo or YouTube Video or Webpage URL)\nai-context https://www.youtube.com/watch?v=video_id\n\n# Make a list of paths\ncat \u003c\u003c EOF \u003e listfile\n../notif\n/working/cybernest\nhttps://github.com/assetnote/h2csmuggler\nhttps://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html\nEOF\n\n# Process URL list concurrently\nai-context -f listfile\n\n# Process private GitHub repository\nGH_TOKEN=$(cat /secrets/GH.PAT) ai-context -u https://github.com/ORG/REPO\n```\n\n\u003e [!WARNING]\n\u003e For directory path (in URL or listfile mode), the path should either start with `/` (absolute) or with `./` or `../` (relative). For current directory, always use `./` for correct regex matching.\n\n### Output\n\n- The tool creates a local folder called `context` and puts everything converted into `.md` files in that folder\n- The filenames have the syntax of `TYPE-PATHNAME.md` (example, `gh-ffuf_ffuf.md`)\n- Every single path in the `listfile` mode will result in a new context file\n- All images (only downloaded via webpages) are named as UUIDs and stored in the `context/images` directory (images are downloaded as a conenience, but doesn't take away from text-first context creation)\n\n### Command Line Options\n\n- CLI argument: provide a path (GitHub repo, YouTube video, WebPage link, or relative/absolute directory path) to process\n- `-f, --file`: provide a file with a list of paths (URLs or directory paths) to process\n- `-i, --ignore`: add additional patterns to ignore during processing (comma-separated)\n- `-t, --threads`: (*optional*) number of workers for concurrent file processing when passing list file (default = 10)\n\n\u003e [!TIP]\n\u003e - Do a `head -n 200 context/FILE.md` (or 500 lines) to view the content tree of the processed code base or directory to see what's been included. Then refine your `-i` flag arguments to ignore additional patterns.\n\u003e - When processing a large number of items, it can look stalled due to thread limits and image download times; use `--debug` to enable verbose logs to know what's running.\n\n### Default Ignores\n\nThe tool includes pre-defined and sensible ignore patterns, including common files and directories that typically don't add value to the context. These are:\n\n- Version control files (.git, .gitignore)\n- Dependencies (node_modules, vendor)\n- Compiled files (*.exe, *.dll)\n- Media files (images, videos, audio)\n- Lock files (package-lock.json, yarn.lock)\n- Build artifacts and caches\n\nFor a full list, see `aicontext/ignores.go`.\n\n## Acknowledgments\n\nThis project takes inspiration from, uses, or references:\n\n- [repomix](https://github.com/yamadashy/repomix): inspiration for turning code into context\n- [innertube](https://github.com/tombulled/innertube): inspiration for code to get transcript from YouTube video\n- [html-to-markdown](https://github.com/JohannesKaufmann/html-to-markdown/v2): used to convert HTML to MD\n- [go-git](https://github.com/go-git/go-git/tree/main): git operations in Go\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftanq16%2Fai-context","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftanq16%2Fai-context","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftanq16%2Fai-context/lists"}