{"id":20188904,"url":"https://github.com/sammcj/ingest","last_synced_at":"2025-04-05T03:02:15.583Z","repository":{"id":250676950,"uuid":"834978333","full_name":"sammcj/ingest","owner":"sammcj","description":"Parse files (e.g. code repos) and websites to clipboard or a file for ingestions by AI / LLMs","archived":false,"fork":false,"pushed_at":"2025-01-30T23:57:45.000Z","size":2821,"stargazers_count":191,"open_issues_count":6,"forks_count":10,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-29T02:01:51.283Z","etag":null,"topics":["ai","codegen","ingest","llama","llm","ollama"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sammcj.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"sammcj"}},"created_at":"2024-07-28T22:07:54.000Z","updated_at":"2025-03-28T01:38:03.000Z","dependencies_parsed_at":"2024-08-26T08:03:52.184Z","dependency_job_id":"cc32bb48-6c23-4faa-8565-9981852a1046","html_url":"https://github.com/sammcj/ingest","commit_stats":null,"previous_names":["sammcj/ingest"],"tags_count":62,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sammcj%2Fingest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sammcj%2Fingest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sammcj%2Fingest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sammcj%2Fingest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sammcj","download_url":"https://codeload.github.com/sammcj/ingest/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247280190,"owners_count":20912966,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","codegen","ingest","llama","llm","ollama"],"created_at":"2024-11-14T03:32:37.897Z","updated_at":"2025-04-05T03:02:15.553Z","avatar_url":"https://github.com/sammcj.png","language":"Go","funding_links":["https://github.com/sponsors/sammcj"],"categories":["Go"],"sub_categories":[],"readme":"# Ingest\n\n![](./ingest-logo-400.png)\n\nIngest parses directories of plain text files, such as source code, into a single markdown file suitable for ingestion by AI/LLMs.\n\n---\n\n![ingest](screenshot2.png)\n\nIngest can also pass the prompt directly to an LLM such as Ollama for processing.\n\n![ingest with --llm](screenshot.png)\n\nAnd ingest web URLs.\n\n![ingest with --web](screenshot3.png)\n\n## Features\n\n- Traverse directory structures and generate a tree view\n- Include/exclude files based on glob patterns\n- Estimate vRAM requirements and check model compatibility using another package I've created called [quantest](https://github.com/sammcj/quantest)\n- Parse output directly to LLMs such as Ollama or any OpenAI compatible API for processing\n- Generate and include git diffs and logs\n- Count approximate tokens for LLM compatibility\n- Customisable output templates\n- Copy output to clipboard (when available)\n- Export to file or print to console\n- Optional JSON output\n- Optionally save output to a file in ~/ingest\n- Shell completions for Bash, Zsh, and Fish\n- Web crawling to ingest web pages as Markdown\n- PDF to markdown conversion and ingestion\n\nIngest Intro (\"Podcast\" Episode):\n\n\u003caudio src=\"https://github.com/sammcj/smcleod_files/raw/refs/heads/master/audio/podcast-ep-sw/Podcast%20Episode%20-%20Ingest.mp3\" controls preload\u003e\u003c/audio\u003e\n\n## Installation\n\n### go install (recommended)\n\nMake sure you have Go installed on your system, then run:\n\n```shell\ngo install github.com/sammcj/ingest@HEAD\n```\n\n### curl\n\nI don't recommend this method as it's not as easy to update, but you can use the following command:\n\n```shell\ncurl -sL https://raw.githubusercontent.com/sammcj/ingest/refs/heads/main/scripts/install.sh | bash\n```\n\n### Manual install\n\n1. Download the latest release from the [releases page](https://github.com/sammcj/ingest/releases)\n2. Move the binary to a directory in your PATH, e.g. `mv ingest* /usr/local/bin/ingest`\n\n## Usage\n\nBasic usage:\n\n```shell\ningest [flags] \u003cpaths\u003e\n```\n\ningest will default to the current working directory if no path is provided, e.g:\n\n```shell\n$ ingest\n\n⠋ Traversing directory and building tree...  [0s]\n[ℹ️] Tokens (Approximate): 15,945\n[✅] Copied to clipboard successfully.\n```\n\nThe first time ingest runs, it will download a small [tokeniser](https://github.com/pkoukk/tiktoken-go-loader/blob/main/assets/cl100k_base.tiktoken) called 'cl100k_base.tiktoken' this is used for tokenisation.\n\nGenerate a prompt from a directory, including only Python files:\n\n```shell\ningest -i \"**/*.py\" /path/to/project\n```\n\nGenerate a prompt with git diff and copy to clipboard:\n\n```shell\ningest -d /path/to/project\n```\n\nGenerate a prompt for multiple files/directories:\n\n```shell\ningest /path/to/project /path/to/other/project\n```\n\nGenerate a prompt and save to a file:\n\n```shell\ningest -o output.md /path/to/project\n```\n\nYou can also provide individual files or multiple paths:\n\n```shell\ningest /path/to/file /path/to/directory\n```\n\nSave output to to ~/ingest/\u003cdirectory_name\u003e.md:\n\n```shell\ningest --save /path/to/project\n```\n\n### VRAM Estimation and Model Compatibility\n\nIngest includes a feature to estimate VRAM requirements and check model compatibility using the [Gollama](https://github.com/sammcj/gollama)'s vramestimator package. This helps you determine if your generated content will fit within the specified model, VRAM, and quantisation constraints.\n\nTo use this feature, add the following flags to your ingest command:\n\n```shell\ningest --vram --model \u003cmodel_id\u003e [--memory \u003cmemory_in_gb\u003e] [--quant \u003cquantisation\u003e] [--context \u003ccontext_length\u003e] [--kvcache \u003ckv_cache_quant\u003e] [--quanttype \u003cquant_type\u003e] [other flags] \u003cpaths\u003e\n```\n\nExamples:\n\nEstimate VRAM usage for a specific context:\n\n```shell\ningest --vram --model NousResearch/Hermes-2-Theta-Llama-3-8B --quant q4_k_m --context 2048 --kvcache q4_0 .\n# Estimated VRAM usage: 5.35 GB\n```\n\nCalculate maximum context for a given memory constraint:\n\n```shell\ningest --vram --model NousResearch/Hermes-2-Theta-Llama-3-8B --quant q4_k_m --memory 6 --kvcache q8_0 .\n# Maximum context for 6.00 GB of memory: 5069\n```\n\nFind the best BPW (Bits Per Weight):\n\n```shell\ningest --vram --model NousResearch/Hermes-2-Theta-Llama-3-8B --memory 6 --quanttype gguf .\n# Best BPW for 6.00 GB of memory: IQ3_S\n```\n\nThe tool also works for exl2 (ExllamaV2) models:\n\n```shell\ningest --vram --model NousResearch/Hermes-2-Theta-Llama-3-8B --quant 5.0 --context 2048 --kvcache q4_0 . # For exl2 models\ningest --vram --model NousResearch/Hermes-2-Theta-Llama-3-8B --quant 5.0 --memory 6 --kvcache q8_0 . # For exl2 models\n```\n\nWhen using the VRAM estimation feature along with content generation, ingest will provide information about the generated content's compatibility with the specified constraints:\n\n```shell\ningest --vram --model NousResearch/Hermes-2-Theta-Llama-3-8B --memory 8 --quant q4_0 .\n⠋ Traversing directory and building tree... [0s]\n[ℹ️] 14,702 Tokens (Approximate)\n[ℹ️] Maximum context for 8.00 GB of memory: 10240\n[✅] Generated content (14,702 tokens) fits within maximum context.\nTop 15 largest files (by estimated token count):\n1. /Users/samm/git/sammcj/ingest/main.go (4,682 tokens)\n2. /Users/samm/git/sammcj/ingest/filesystem/filesystem.go (2,694 tokens)\n3. /Users/samm/git/sammcj/ingest/README.md (1,895 tokens)\n4. /Users/samm/git/sammcj/ingest/utils/utils.go (948 tokens)\n5. /Users/samm/git/sammcj/ingest/config/config.go (884 tokens)\n[✅] Copied to clipboard successfully.\n```\n\nAvailable flags for VRAM estimation:\n\n- `--vram`: Enable VRAM estimation and model compatibility check\n- `--model`: Specify the model ID to check against (required for estimation)\n- `--memory`: Specify the available memory in GB for context calculation (optional)\n- `--quant`: Specify the quantisation type (e.g., q4_k_m) or bits per weight (e.g., 5.0)\n- `--context`: Specify the context length for VRAM estimation (optional)\n- `--kvcache`: Specify the KV cache quantisation (fp16, q8_0, or q4_0)\n- `--quanttype`: Specify the quantisation type (gguf or exl2)\n\nIngest will provide appropriate output based on the combination of flags used, such as estimating VRAM usage, calculating maximum context, or finding the best BPW. If the generated content fits within the specified constraints, you'll see a success message. Otherwise, you'll receive a warning that the content may not fit.\n\n## LLM Integration\n\nIngest can pass the generated prompt to LLMs that have an OpenAI compatible API such as [Ollama](https://ollama.com) for processing.\n\n```shell\ningest --llm /path/to/project\n```\n\nBy default this will use any prompt suffix from your configuration file:\n\n```shell\n./ingest utils.go --llm\n⠋ Traversing directory and building tree...  [0s]\nThis is Go code for a file named `utils.go`. It contains various utility functions for\nhandling terminal output, clipboard operations, and configuration directories.\n...\n```\n\nYou can provide a prompt suffix to append to the generated prompt:\n\n```shell\ningest --llm -p \"explain this code\" /path/to/project\n```\n\n## Web Crawling \u0026 Ingestion\n\nCrawl with explicit web mode\n\n```shell\ningest --web https://example.com\n```\n\nAuto-detect URL and crawl\n\n```shell\ningest https://example.com\n```\n\nCrawl with domain restriction\n\n```shell\ningest --web --web-domains example.com https://example.com\n```\n\nCrawl deeper with more concurrency\n\n```shell\ningest --web --web-depth 3 --web-concurrent 10 https://example.com\n```\n\nExclude a path from the crawl\n\n```shell\ningest --web https://example.com -e '/posts/**'\n```\n\n## Shell Completions\n\nIngest includes shell completions for Bash, Zsh, and Fish.\n\n```shell\nsource \u003c(ingest completion zsh)\n```\n\nSee `ingest completion -h` for more information.\n\n## Configuration\n\nIngest uses a configuration file located at `~/.config/ingest/ingest.json`.\n\nYou can make Ollama processing run without prompting setting `\"llm_auto_run\": true` in the config file.\n\nThe config file also contains:\n\n- `llm_model`: The model to use for processing the prompt, e.g. \"llama3.1:8b-q5_k_m\".\n- `llm_prompt_prefix`: An optional prefix to prepend to the prompt, e.g. \"This is my application.\"\n- `llm_prompt_suffix`: An optional suffix to append to the prompt, e.g. \"explain this code\"\n\nIngest uses the following directories for user-specific configuration:\n\n- `~/.config/ingest/patterns/exclude`: Add .glob files here to exclude additional patterns.\n- `~/.config/ingest/patterns/templates`: Add custom .tmpl files here for different output formats.\n\nThese directories will be created automatically on first run, along with README files explaining their purpose.\n\n### Flags\n\n- `--config`: Opens the config file in the default editor\n- `--context`: Specify the context length for VRAM estimation\n- `--exclude-from-tree`: Exclude files/folders from the source tree based on exclude patterns\n- `--git-diff-branch`: Generate git diff between two branches\n- `--git-log-branch`: Retrieve git log between two branches\n- `--include-priority`: Include files in case of conflict between include and exclude patterns\n- `--json`: Print output as JSON\n- `--kvcache`: Specify the KV cache quantisation\n- `--llm`: Send the generated prompt to an OpenAI compatible LLM server (such as Ollama) for processing\n- `--memory`: Specify the available memory in GB for context calculation\n- `--model`: Specify the model ID for VRAM estimation\n- `--no-codeblock`: Disable wrapping code inside markdown code blocks\n- `--no-default-excludes`: Disable default exclude patterns\n- `--pattern-exclude`: Path to a specific .glob file for exclude patterns\n- `--print-default-excludes`: Print the default exclude patterns\n- `--print-default-template`: Print the default template\n- `--quant`: Specify the quantisation type or bits per weight\n- `--quanttype`: Specify the quantisation type (gguf or exl2)\n- `--relative-paths`: Use relative paths instead of absolute paths\n- `--report`: Print the largest parsed files\n- `--save`: Save output to ~/ingest/\u003cdirectory_name\u003e.md\n- `--tokens`: Display the token count of the generated prompt\n- `--verbose`: Print verbose output\n- `--vram`: Estimate VRAM usage and check model compatibility\n- `--web-concurrent`: Maximum concurrent requests for web crawling\n- `--web-depth`: Maximum depth for web crawling\n- `--web-domains`: Comma-separated list of domains to restrict web crawling\n- `--web`: Crawl a web page\n- `-c, --encoding`: Optional tokeniser to use for token count\n- `-d, --diff`: Include git diff\n- `-e, --exclude`: Patterns to exclude (can be used multiple times)\n- `-i, --include`: Patterns to include (can be used multiple times)\n- `-l, --line-number`: Add line numbers to the source code\n- `-n, --no-clipboard`: Disable copying to clipboard\n- `-o, --output`: Optional output file path\n- `-p, --prompt`: Optional prompt suffix to append to the generated prompt\n- `-t, --template`: Path to a custom Handlebars template\n- `-V, --version`: Print the version number (WIP - still trying to get this to work nicely)\n\n### Excludes\n\nYou can get a list of the default excludes by parsing `--print-default-excludes` to ingest.\nThese are defined in [defaultExcludes.go](https://github.com/sammcj/ingest/blob/main/filesystem/defaultExcludes.go).\n\nTo override the default excludes, create a `default.glob` file in `~/.config/ingest/patterns/exclude` with the patterns you want to exclude.\n\n### Templates\n\nTemplates are written in standard [go templating syntax](https://pkg.go.dev/text/template).\n\nYou can get a list of the default templates by parsing `--print-default-template` to ingest.\nThese are defined in [template.go](https://github.com/sammcj/ingest/blob/main/template/template.go).\n\nTo override the default templates, create a `default.tmpl` file in `~/.config/ingest/patterns/templates` with the template you want to use by default.\n\n## Contributing\n\nContributions are welcome, Please feel free to submit a Pull Request.\n\n## License\n\n- Copyright 2024 Sam McLeod\n- This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## Acknowledgements\n\n- Initially inspired by [mufeedvh/code2prompt](https://github.com/mufeedvh/code2prompt)\n\n\u003cscript src=\"http://api.html5media.info/1.1.8/html5media.min.js\"\u003e\u003c/script\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsammcj%2Fingest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsammcj%2Fingest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsammcj%2Fingest/lists"}