{"id":13524929,"url":"https://github.com/ludovicianul/hq","last_synced_at":"2026-01-24T05:07:52.673Z","repository":{"id":151308551,"uuid":"404008263","full_name":"ludovicianul/hq","owner":"ludovicianul","description":"lightweight command line HTML processor using CSS and XPath selectors","archived":false,"fork":false,"pushed_at":"2024-09-19T13:56:00.000Z","size":109,"stargazers_count":67,"open_issues_count":0,"forks_count":3,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-11-02T09:32:08.817Z","etag":null,"topics":["cli","command-line","css-selectors","graal-native","hacktoberfest","html","java","jq","xpath"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ludovicianul.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-07T14:25:08.000Z","updated_at":"2024-09-30T21:12:03.000Z","dependencies_parsed_at":"2023-09-24T13:19:00.030Z","dependency_job_id":"e4fcec59-4653-4377-96f4-5fa73cc6387d","html_url":"https://github.com/ludovicianul/hq","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ludovicianul%2Fhq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ludovicianul%2Fhq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ludovicianul%2Fhq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ludovicianul%2Fhq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ludovicianul","download_url":"https://codeload.github.com/ludovicianul/hq/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246585506,"owners_count":20801022,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","command-line","css-selectors","graal-native","hacktoberfest","html","java","jq","xpath"],"created_at":"2024-08-01T06:01:14.679Z","updated_at":"2026-01-24T05:07:52.629Z","avatar_url":"https://github.com/ludovicianul.png","language":"Shell","funding_links":[],"categories":["Command Line","Shell"],"sub_categories":["Like jq"],"readme":"# hq\n\nSmall utility to parse and grep HTML files. It\nuses [CSS selectors](https://www.w3schools.com/cssref/css_selectors.asp) or [XPath Selectors](https://www.w3schools.com/xml/xpath_intro.asp) to extract HTML elements.\n\n# Usage\n\n```bash\nhq - command line HTML elements finder; version 1.0.0\n\nUsage: hq [-hptV] [-a=\u003cattribute\u003e] [-f=\u003cFILE\u003e] [-o=\u003cFILE\u003e] [-s=\u003cPOLICY\u003e] [-x=\u003cXPATH\u003e] \u003cselector\u003e\n          [COMMAND]\n      \u003cselector\u003e            The CSS selector\n  -a, --attribute=\u003cattribute\u003e\n                            Return only this attribute from the selected HTML elements\n  -f, --file=\u003cFILE\u003e         The HTML input file. If not supplied it will default to stdin\n  -h, --help                Show this help message and exit.\n  -o, --output=\u003cFILE\u003e       The output file. If not supplied it will default to stdout\n  -p, --pretty              Force pretty printing the output\n  -r, --remove=\u003cSELECTOR\u003e   Remove nodes matching given selector\n  -s, --sanitize=\u003cPOLICY\u003e   Sanitizes the html input according to the given policy\n  -t, --text                Display only the inner text of the selected HTML top element\n  -V, --version             Print version information and exit.\n  -x, --xpath=\u003cXPATH\u003e       Supply an XPath selector instead of CSS\nCommands:\n  generate-completion  Generate bash/zsh completion script for hq.\n\n```\n\n# Installation\n\n## Homebrew\n\n```\n\u003e brew tap ludovicianul/tap\n\u003e brew install ludovicianul/tap/hq\n```\n\n## Manual\n\n`hq` is compiled to native code using GraalVM. Check\nthe [release page](https://github.com/ludovicianul/hq/releases/) for binaries (Linux,\nMacOS, uberjar).\n\nAfter download, you can make `hq` globally available:\n\n```bash\nsudo cp hq-macos /usr/local/bin/hq\n```\n\nThe uberjar can be run using `java -jar hq`. Requires Java 11+.\n\n# Autocomplete\nRun the following commands to get autocomplete:\n\n```bash\nhq generate-completion \u003e\u003e hq_autocomplete\n\nsource hq_autocomplete\n```\n\n# HTML Sanitizing\n`hq` can sanitize html output. Supported modes are: `NONE, BASIC, SIMPLE_TEXT, BASIC_WITH_IMAGES, RELAXED`. \n\nThis is how sanitization works: \n\n| Policy | Details |\n| ------- | ------- | \n| `NONE` | Allows only text nodes: all HTML will be stripped. |\n| `BASIC` | Allows a fuller range of text nodes: `a, b, blockquote, br, cite, code, dd, dl, dt, em, i, li, ol, p, pre, q, small, span, strike, strong, sub, sup, u, ul`, and appropriate attributes. Does not allow images.|\n| `SIMPLE_TEXT` | Allows only simple text formatting: `b, em, i, strong, u`. All other HTML (tags and attributes) will be removed.| \n| `BASIC_WITH_IMAGES` | Allows the same text tags as `BASIC`, and also allows `img` tags, with appropriate attributes, with `src` pointing to `http` or `https`.\n| `RELAXES` | Allows a full range of text and structural body HTML: `a, b, blockquote, br, caption, cite, code, col, colgroup, dd, div, dl, dt, em, h1, h2, h3, h4, h5, h6, i, img, li, ol, p, pre, q, small, span, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, u, ul`.|\n\n# Examples\n\nGet the `div` with id `mainLeaderboard`:\n\n```\n➜ curl -s https://www.w3schools.com/cssref/css_selectors.php | hq \"#main \u003e p:nth-child(6)\" -t\n\nIn CSS, selectors are patterns used to select the element(s) you want to style.\n\n```\n\nGet the text inside an article:\n\n```\n➜ curl -s https://ludovicianul.github.io/2021/07/16/unicode_language_version/ | hq '.post' -t\n\nMake sure you know which Unicode version is supported by your programming language version 16 Jul 2021 While enhancing CATS I recently added a feature to send requests that include \nsingle and multi code point emojis. This is a single code point emoji: 🥶, which can be represented in Java as the \\uD83E\\uDD76 string. The test case is simple: inject emojis within \nstrings and expect that the REST endpoint will sanitize the input and remove them entirely (I appreciate this might not be a valid case for all APIs, this is why the behaviour is \nconfigurable in CATS, but not the focus of this article). I usually recommend that any REST endpoint should sanitize input before validating it and remove special characters. \nA typical regex for this would be [\\p{C}\\p{Z}\\p{So}]+ (although you should enhance it to allow spaces between words), which means: p{C} - match Unicode invisible Control \nChars (\\u000D - carriage return for example) ...\n...\n```\n\nSanitize the html according to the [specified policy](#html-sanitizing):\n```\n ➜ curl -s https://ludovicianul.github.io/2021/07/16/unicode_language_version/ | hq html -s=BASIC -p\n\n\u003chtml\u003e\n    \u003chead\u003e\u003c/head\u003e\n    \u003cbody\u003e\n        \u003ca href=\"https://ludovicianul.github.io/\" rel=\"nofollow\"\u003e m's blog \u003c/a\u003e\n        \u003cp\u003epractical thoughts about software engineering\u003c/p\u003e\n        \u003ca href=\"https://ludovicianul.github.io/\" rel=\"nofollow\"\u003eHome\u003c/a\u003e\n        \u003ca rel=\"nofollow\"\u003eAbout\u003c/a\u003e\n        \u003ca href=\"https://github.com/ludovicianul\" rel=\"nofollow\"\u003eGitHub\u003c/a\u003e\n        \u003cp\u003e© 2021. All rights reserved.\u003c/p\u003e\n        Make sure you know which Unicode version is supported by your programming language version\n        \u003cspan\u003e16 Jul 2021\u003c/span\u003e\n        \u003cp\u003e\n...\n    \u003c/body\u003e\n\u003c/html\u003e\n```\n\nGet all `href` attributes from a given page:\n\n```shell\n ➜ curl -s https://ludovicianul.github.io | hq \"*\" -a \"href\"\nhttp://gmpg.org/xfn/11\nhttps://ludovicianul.github.io/public/css/poole.css\nhttps://ludovicianul.github.io/public/css/syntax.css\nhttps://ludovicianul.github.io/public/css/hyde.css\nhttps://fonts.googleapis.com/css?family=PT+Sans:400,400italic,700|Abril+Fatface\nhttps://ludovicianul.github.io/public/apple-touch-icon-144-precomposed.png\nhttps://ludovicianul.github.io/public/favicon.ico\n/atom.xml\nhttps://ludovicianul.github.io/\nhttps://ludovicianul.github.io/\n/about/\n...\n```\n\n# Resources\n\n- [Universal selector in CSS](https://www.scaler.com/topics/universal-selector-in-css/)\n- [HTML elements](https://developer.mozilla.org/en-US/docs/Web/HTML/Element)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fludovicianul%2Fhq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fludovicianul%2Fhq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fludovicianul%2Fhq/lists"}