{"id":13525005,"url":"https://github.com/EricChiang/pup","last_synced_at":"2025-04-01T04:30:38.456Z","repository":{"id":20254382,"uuid":"23527160","full_name":"ericchiang/pup","owner":"ericchiang","description":"Parsing HTML at the command line","archived":false,"fork":false,"pushed_at":"2024-05-02T13:43:38.000Z","size":3865,"stargazers_count":8099,"open_issues_count":105,"forks_count":256,"subscribers_count":89,"default_branch":"master","last_synced_at":"2024-10-14T16:20:59.717Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ericchiang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-09-01T01:31:29.000Z","updated_at":"2024-10-10T14:24:57.000Z","dependencies_parsed_at":"2023-01-13T20:52:17.956Z","dependency_job_id":"703cb31a-852e-4ad4-ad25-51f23955ff04","html_url":"https://github.com/ericchiang/pup","commit_stats":{"total_commits":87,"total_committers":12,"mean_commits":7.25,"dds":"0.13793103448275867","last_synced_commit":"5a57cf111366c7c08999a34b2afd7ba36d58a96d"},"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ericchiang%2Fpup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ericchiang%2Fpup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ericchiang%2Fpup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ericchiang%2Fpup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ericchiang","download_url":"https://codeload.github.com/ericchiang/pup/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222698201,"owners_count":17024878,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T06:01:15.257Z","updated_at":"2024-11-02T09:30:40.388Z","avatar_url":"https://github.com/ericchiang.png","language":"HTML","funding_links":[],"categories":["Command Line"],"sub_categories":["Like jq"],"readme":"# pup\n\npup is a command line tool for processing HTML. It reads from stdin,\nprints to stdout, and allows the user to filter parts of the page using\n[CSS selectors](https://developer.mozilla.org/en-US/docs/Web/Guide/CSS/Getting_started/Selectors).\n\nInspired by [jq](http://stedolan.github.io/jq/), pup aims to be a\nfast and flexible way of exploring HTML from the terminal.\n\n## Install\n\nDirect downloads are available through the [releases page](https://github.com/EricChiang/pup/releases/latest).\n\nIf you have Go installed on your computer just run `go get`.\n\n    go get github.com/ericchiang/pup\n\nIf you're on OS X, use [Homebrew](http://brew.sh/) to install (no Go required).\n\n    brew install https://raw.githubusercontent.com/EricChiang/pup/master/pup.rb\n\n## Quick start\n\n```bash\n$ curl -s https://news.ycombinator.com/\n```\n\nEw, HTML. Let's run that through some pup selectors:\n\n```bash\n$ curl -s https://news.ycombinator.com/ | pup 'table table tr:nth-last-of-type(n+2) td.title a'\n```\n\nOkay, how about only the links?\n\n```bash\n$ curl -s https://news.ycombinator.com/ | pup 'table table tr:nth-last-of-type(n+2) td.title a attr{href}'\n```\n\nEven better, let's grab the titles too:\n\n```bash\n$ curl -s https://news.ycombinator.com/ | pup 'table table tr:nth-last-of-type(n+2) td.title a json{}'\n```\n\n## Basic Usage\n\n```bash\n$ cat index.html | pup [flags] '[selectors] [display function]'\n```\n\n## Examples\n\nDownload a webpage with wget.\n\n```bash\n$ wget http://en.wikipedia.org/wiki/Robots_exclusion_standard -O robots.html\n```\n\n#### Clean and indent\n\nBy default pup will fill in missing tags and properly indent the page.\n\n```bash\n$ cat robots.html\n# nasty looking HTML\n$ cat robots.html | pup --color\n# cleaned, indented, and colorful HTML\n```\n\n#### Filter by tag\n\n```bash\n$ cat robots.html | pup 'title'\n\u003ctitle\u003e\n Robots exclusion standard - Wikipedia, the free encyclopedia\n\u003c/title\u003e\n```\n\n#### Filter by id\n\n```bash\n$ cat robots.html | pup 'span#See_also'\n\u003cspan class=\"mw-headline\" id=\"See_also\"\u003e\n See also\n\u003c/span\u003e\n```\n\n#### Filter by attribute\n\n```bash\n$ cat robots.html | pup 'th[scope=\"row\"]'\n\u003cth scope=\"row\" class=\"navbox-group\"\u003e\n Exclusion standards\n\u003c/th\u003e\n\u003cth scope=\"row\" class=\"navbox-group\"\u003e\n Related marketing topics\n\u003c/th\u003e\n\u003cth scope=\"row\" class=\"navbox-group\"\u003e\n Search marketing related topics\n\u003c/th\u003e\n\u003cth scope=\"row\" class=\"navbox-group\"\u003e\n Search engine spam\n\u003c/th\u003e\n\u003cth scope=\"row\" class=\"navbox-group\"\u003e\n Linking\n\u003c/th\u003e\n\u003cth scope=\"row\" class=\"navbox-group\"\u003e\n People\n\u003c/th\u003e\n\u003cth scope=\"row\" class=\"navbox-group\"\u003e\n Other\n\u003c/th\u003e\n```\n\n#### Pseudo Classes\n\nCSS selectors have a group of specifiers called [\"pseudo classes\"](\nhttps://developer.mozilla.org/en-US/docs/Web/CSS/Pseudo-classes)  which are pretty\ncool. pup implements a majority of the relevant ones them.\n\nHere are some examples.\n\n```bash\n$ cat robots.html | pup 'a[rel]:empty'\n\u003ca rel=\"license\" href=\"//creativecommons.org/licenses/by-sa/3.0/\" style=\"display:none;\"\u003e\n\u003c/a\u003e\n```\n\n```bash\n$ cat robots.html | pup ':contains(\"History\")'\n\u003cspan class=\"toctext\"\u003e\n History\n\u003c/span\u003e\n\u003cspan class=\"mw-headline\" id=\"History\"\u003e\n History\n\u003c/span\u003e\n```\n\n```bash\n$ cat robots.html | pup ':parent-of([action=\"edit\"])'\n\u003cspan class=\"wb-langlinks-edit wb-langlinks-link\"\u003e\n \u003ca action=\"edit\" href=\"//www.wikidata.org/wiki/Q80776#sitelinks-wikipedia\" text=\"Edit links\" title=\"Edit interlanguage links\" class=\"wbc-editpage\"\u003e\n  Edit links\n \u003c/a\u003e\n\u003c/span\u003e\n```\n\nFor a complete list, view the [implemented selectors](#implemented-selectors)\nsection.\n\n\n#### `+`, `\u003e`, and `,`\n\nThese are intermediate characters that declare special instructions. For\ninstance, a comma `,` allows pup to specify multiple groups of selectors.\n\n```bash\n$ cat robots.html | pup 'title, h1 span[dir=\"auto\"]'\n\u003ctitle\u003e\n Robots exclusion standard - Wikipedia, the free encyclopedia\n\u003c/title\u003e\n\u003cspan dir=\"auto\"\u003e\n Robots exclusion standard\n\u003c/span\u003e\n```\n\n#### Chain selectors together\n\nWhen combining selectors, the HTML nodes selected by the previous selector will\nbe passed to the next ones.\n\n```bash\n$ cat robots.html | pup 'h1#firstHeading'\n\u003ch1 id=\"firstHeading\" class=\"firstHeading\" lang=\"en\"\u003e\n \u003cspan dir=\"auto\"\u003e\n  Robots exclusion standard\n \u003c/span\u003e\n\u003c/h1\u003e\n```\n\n```bash\n$ cat robots.html | pup 'h1#firstHeading span'\n\u003cspan dir=\"auto\"\u003e\n Robots exclusion standard\n\u003c/span\u003e\n```\n\n## Implemented Selectors\n\nFor further examples of these selectors head over to [MDN](\nhttps://developer.mozilla.org/en-US/docs/Web/CSS/Reference).\n\n```bash\npup '.class'\npup '#id'\npup 'element'\npup 'selector + selector'\npup 'selector \u003e selector'\npup '[attribute]'\npup '[attribute=\"value\"]'\npup '[attribute*=\"value\"]'\npup '[attribute~=\"value\"]'\npup '[attribute^=\"value\"]'\npup '[attribute$=\"value\"]'\npup ':empty'\npup ':first-child'\npup ':first-of-type'\npup ':last-child'\npup ':last-of-type'\npup ':only-child'\npup ':only-of-type'\npup ':contains(\"text\")'\npup ':nth-child(n)'\npup ':nth-of-type(n)'\npup ':nth-last-child(n)'\npup ':nth-last-of-type(n)'\npup ':not(selector)'\npup ':parent-of(selector)'\n```\n\nYou can mix and match selectors as you wish.\n\n```bash\ncat index.html | pup 'element#id[attribute=\"value\"]:first-of-type'\n```\n\n## Display Functions\n\nNon-HTML selectors which effect the output type are implemented as functions\nwhich can be provided as a final argument.\n\n#### `text{}`\n\nPrint all text from selected nodes and children in depth first order.\n\n```bash\n$ cat robots.html | pup '.mw-headline text{}'\nHistory\nAbout the standard\nDisadvantages\nAlternatives\nExamples\nNonstandard extensions\nCrawl-delay directive\nAllow directive\nSitemap\nHost\nUniversal \"*\" match\nMeta tags and headers\nSee also\nReferences\nExternal links\n```\n\n#### `attr{attrkey}`\n\nPrint the values of all attributes with a given key from all selected nodes.\n\n```bash\n$ cat robots.html | pup '.catlinks div attr{id}'\nmw-normal-catlinks\nmw-hidden-catlinks\n```\n\n#### `json{}`\n\nPrint HTML as JSON.\n\n```bash\n$ cat robots.html  | pup 'div#p-namespaces a'\n\u003ca href=\"/wiki/Robots_exclusion_standard\" title=\"View the content page [c]\" accesskey=\"c\"\u003e\n Article\n\u003c/a\u003e\n\u003ca href=\"/wiki/Talk:Robots_exclusion_standard\" title=\"Discussion about the content page [t]\" accesskey=\"t\"\u003e\n Talk\n\u003c/a\u003e\n```\n\n```bash\n$ cat robots.html | pup 'div#p-namespaces a json{}'\n[\n {\n  \"accesskey\": \"c\",\n  \"href\": \"/wiki/Robots_exclusion_standard\",\n  \"tag\": \"a\",\n  \"text\": \"Article\",\n  \"title\": \"View the content page [c]\"\n },\n {\n  \"accesskey\": \"t\",\n  \"href\": \"/wiki/Talk:Robots_exclusion_standard\",\n  \"tag\": \"a\",\n  \"text\": \"Talk\",\n  \"title\": \"Discussion about the content page [t]\"\n }\n]\n```\n\nUse the `-i` / `--indent` flag to control the intent level.\n\n```bash\n$ cat robots.html | pup -i 4 'div#p-namespaces a json{}'\n[\n    {\n        \"accesskey\": \"c\",\n        \"href\": \"/wiki/Robots_exclusion_standard\",\n        \"tag\": \"a\",\n        \"text\": \"Article\",\n        \"title\": \"View the content page [c]\"\n    },\n    {\n        \"accesskey\": \"t\",\n        \"href\": \"/wiki/Talk:Robots_exclusion_standard\",\n        \"tag\": \"a\",\n        \"text\": \"Talk\",\n        \"title\": \"Discussion about the content page [t]\"\n    }\n]\n```\n\nIf the selectors only return one element the results will be printed as a JSON\nobject, not a list.\n\n```bash\n$ cat robots.html  | pup --indent 4 'title json{}'\n{\n    \"tag\": \"title\",\n    \"text\": \"Robots exclusion standard - Wikipedia, the free encyclopedia\"\n}\n```\n\nBecause there is no universal standard for converting HTML/XML to JSON, a\nmethod has been chosen which hopefully fits. The goal is simply to get the\noutput of pup into a more consumable format.\n\n## Flags\n\nRun `pup --help` for a list of further options\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FEricChiang%2Fpup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FEricChiang%2Fpup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FEricChiang%2Fpup/lists"}