{"id":13596081,"url":"https://github.com/ColonelBuendia/rgpipe","last_synced_at":"2025-04-09T16:31:31.180Z","repository":{"id":36437657,"uuid":"221604259","full_name":"ColonelBuendia/rgpipe","owner":"ColonelBuendia","description":"lesspipe for ripgrep for common new filetypes using few dependencies ","archived":false,"fork":false,"pushed_at":"2022-01-24T05:08:21.000Z","size":62,"stargazers_count":31,"open_issues_count":0,"forks_count":5,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-11-06T19:40:52.808Z","etag":null,"topics":["ebooks","excel","full-text-search","indexing","office","pdf","powerpoint","ripgrep","search","word"],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ColonelBuendia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-11-14T03:32:43.000Z","updated_at":"2024-02-08T18:54:34.000Z","dependencies_parsed_at":"2022-09-16T11:50:13.840Z","dependency_job_id":null,"html_url":"https://github.com/ColonelBuendia/rgpipe","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ColonelBuendia%2Frgpipe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ColonelBuendia%2Frgpipe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ColonelBuendia%2Frgpipe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ColonelBuendia%2Frgpipe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ColonelBuendia","download_url":"https://codeload.github.com/ColonelBuendia/rgpipe/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248067768,"owners_count":21042352,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ebooks","excel","full-text-search","indexing","office","pdf","powerpoint","ripgrep","search","word"],"created_at":"2024-08-01T16:02:07.550Z","updated_at":"2025-04-09T16:31:30.813Z","avatar_url":"https://github.com/ColonelBuendia.png","language":"Shell","funding_links":[],"categories":["Shell"],"sub_categories":[],"readme":"`rgpipe` is a single bash/sh script and an alias to use with ripgrep to search through a myriad of file types that are otherwise not grep friendly.  Use it with ripgrep's -pre command which allows ripgrep to selectively process files before searching.\n\n\n# TL;DR\n\nThe most basic usage is to point `rgpipe` at some file, and it will attempt to print the contents of said file to stdout. \n\n`rgpipe MyFancyExcelFile.xlsx`\n\nThe more involved usage is as a filter in front of [ripgrep](https://github.com/BurntSushi/ripgrep) to systematically attempt to grep through the contents of assorted non-text files much as you would text files. The basic incantation looks like: \n\n`rg --pre-glob '*.{xlsx,pptx,docx,pdf}' --pre rgpipe \"$YourSearchTermHere\"`\n\n## Overview\n\nI wrote up an extended gist about how to use it\n[here](https://gist.github.com/ColonelBuendia/314826e37ec35c616d70506c38dc65aa)\n\nThat gist is only useful because of the kind note by BurntSushi in [this hacker news comment](https://news.ycombinator.com/item?id=19675934) explaining how `rg --pre-glob` works.\n\n**This helps grep through:**\n\n- New MS Office files (DOCX, PPTX, XLSX, variants thereof)  \n    + Uses `unzip` and `sed`\n- Old MS Office files (DOC, PPT, XLS, variants thereof) \u0026 new excel binary format\n    + Uses `strings`\n- LibreOffice files (ODS, ODT, ODP)  \n    + Uses `unzip` and `sed`\n- PDF  \n    + Uses `pdftottext` from poppler\n- Web/structured formats (HTML, XHTML ...)  \n    + Uses `w3m` lynx and friends also works.  Not 100% necessary.  \n- Web formats disguised as books (chm, epub)  \n    + `unzip` and `w3m` for EPUB\n    + `7zip` and `w3m` for chm\n\n\n#### Specifically\n\nUbuntu wants: `sudo apt install poppler-utils p7zip w3m unzip`\n\ntermux wants: `pkg install poppler p7zip w3m`\n\n\n## Usage notes\n\n### Vanilla ripgrep usage\n\nAssuming rgpipe is in path, use /path/to/rgpipe if it's not\n\n```bash\nrg --pre rgpipe YourSearchTermHere\n```\n\n### Better ripgrep usage\n\nAbove uses rgpipe even when it's not needed, which is slow, ripgrep can selectively use it with --pre-glob\n\n```bash\nrg --pre-glob '*.{xlsx,pptx,docx,pdf}' --pre rgpipe YourSearchTermHere\n```\n\nA more thorough pre glob:\n\n```bash\nrg --pre-glob '*.{pdf,xl[tas][bxm],xl[wsrta],do[ct],do[ct][xm],p[po]t[xm],p[op]t,html,htm,xhtm,xhtml,epub,chm,od[stp]}' --pre rgpipe YourSearchTermHere\n```\n\nAn alias because that is a lot of typing\n\n```bash\nalias rgg=\"rg -i -z --max-columns-preview --max-columns 500 --hidden --no-ignore --pre-glob \\\n'*.{pdf,xl[tas][bxm],xl[wsrta],do[ct],do[ct][xm],p[po]t[xm],p[op]t,html,htm,xhtm,xhtml,epub,chm,od[stp]}' --pre rgpipe\"\n```\n\n### Poor man's full text search\n\nStep 1: use rgpipe to make text sidecar files\n\n```bash\nfind-rgpipe-type() {\n     find `pwd` -type f -iname \"*.$1\" -exec sh -c 'for f; do rgpipe \"$f\" \u003e \"${f%.*}.txt\"; done' _ {} +\n}\n\n# or get fancy with xargs for multithreaded goodness\n\nfind-rgpipe-type-xargs() {\n    find \"$(pwd)\" -type f -iname \"*.$1\" -print0 | xargs -0 -P0 -n 1 -I {} sh -c 'rgpipe \"{}\" \u003e \"{}.txt\"'\n}\n\n```\n\nMake text sidecars for all files with PDF extension under current directory using the function defined above.  \n\n```bash\nfind-rgpipe-type pdf\n```\n\nStep 2: Use ripgrep to search those files \n\n```bash\nrg YourSearchTermHere\n```\n\n## Super useful\n\n1 - [this hacker news comment](https://news.ycombinator.com/item?id=19675934)\n\n2 - [The pre processing script](http://manpages.ubuntu.com/manpages/cosmic/man1/rg.1.html) that is the template into which I added some more file types\n\n3 - midnight commander has [great scripts](https://github.com/MidnightCommander/mc/tree/0075f36b693109a34f8729d47fdf931b0481a68e/misc/ext.d) on this subject\n\n4 - [lesspipe of course](https://github.com/wofr06/lesspipe)\n\n5 - [rga](https://github.com/phiresky/ripgrep-all) is a rust based tool doing a similar thing\n\n## The name\n\n`rgpipe` because the idea is similar to lesspipe.  \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FColonelBuendia%2Frgpipe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FColonelBuendia%2Frgpipe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FColonelBuendia%2Frgpipe/lists"}