{"id":13579386,"url":"https://github.com/simonw/strip-tags","last_synced_at":"2025-04-05T16:04:37.818Z","repository":{"id":166836748,"uuid":"642453139","full_name":"simonw/strip-tags","owner":"simonw","description":"CLI tool for stripping tags from HTML","archived":false,"fork":false,"pushed_at":"2023-12-19T14:26:47.000Z","size":40,"stargazers_count":206,"open_issues_count":7,"forks_count":6,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-10-18T07:53:44.007Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/simonw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-05-18T15:44:34.000Z","updated_at":"2024-10-13T23:41:33.000Z","dependencies_parsed_at":"2024-01-16T20:29:19.110Z","dependency_job_id":"df207df6-a452-4184-933b-068c36e5bbe0","html_url":"https://github.com/simonw/strip-tags","commit_stats":null,"previous_names":["simonw/strip-tags"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fstrip-tags","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fstrip-tags/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fstrip-tags/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fstrip-tags/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/simonw","download_url":"https://codeload.github.com/simonw/strip-tags/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247361615,"owners_count":20926642,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T15:01:38.941Z","updated_at":"2025-04-05T16:04:37.794Z","avatar_url":"https://github.com/simonw.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# strip-tags\n\n[![PyPI](https://img.shields.io/pypi/v/strip-tags.svg)](https://pypi.org/project/strip-tags/)\n[![Changelog](https://img.shields.io/github/v/release/simonw/strip-tags?include_prereleases\u0026label=changelog)](https://github.com/simonw/strip-tags/releases)\n[![Tests](https://github.com/simonw/strip-tags/workflows/Test/badge.svg)](https://github.com/simonw/strip-tags/actions?query=workflow%3ATest)\n[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/strip-tags/blob/master/LICENSE)\n\nStrip tags from HTML, optionally from areas identified by CSS selectors\n\nSee [llm, ttok and strip-tags—CLI tools for working with ChatGPT and other LLMs](https://simonwillison.net/2023/May/18/cli-tools-for-llms/) for more on this project.\n\n## Installation\n\nInstall this tool using `pip`:\n```bash\npip install strip-tags\n```\n## Usage\n\nPipe content into this tool to strip tags from it:\n```bash\ncat input.html | strip-tags \u003e output.txt\n````\nOr pass a filename:\n```bash\nstrip-tags -i input.html \u003e output.txt\n```\nTo run against just specific areas identified by CSS selectors:\n```bash\nstrip-tags '.content' -i input.html \u003e output.txt\n```\nThis can be called with multiple selectors:\n```bash\ncat input.html | strip-tags '.content' '.sidebar' \u003e output.txt\n```\nTo return just the first element on the page that matches one of the selectors, use `--first`:\n```bash\ncat input.html | strip-tags .content --first \u003e output.txt\n```\nTo remove content contained by specific selectors - e.g. the `\u003cnav\u003e` section of a page, use `-r` or `--remove`:\n```bash\ncat input.html | strip-tags -r nav \u003e output.txt\n```\nTo minify whitespace - reducing multiple space and tab characters to a single space, removing any remaining blank lines - add `-m` or `--minify`:\n```bash\ncat input.html | strip-tags -m \u003e output.txt\n```\nYou can also run this command using `python -m` like this:\n```bash\npython -m strip_tags --help\n```\n### Keeping the markup for specified tags\n\nWhen passing content to a language model, it can sometimes be useful to leave in a subset of HTML tags - `\u003ch1\u003eThis is the heading\u003c/h1\u003e` for example - to provide extra hints to the model.\n\nThe `-t/--keep-tag` option can be passed multiple times to specify tags that should be kept.\n\nThis example looks at the `\u003cheader\u003e` section of https://datasette.io/ and keeps the tags around the list items and `\u003ch1\u003e` elements:\n\n```\ncurl -s https://datasette.io/ | strip-tags header -t h1 -t li\n```\n```html\n\u003cli\u003eUses\u003c/li\u003e\n\u003cli\u003eDocumentation Docs\u003c/li\u003e\n\u003cli\u003eTutorials\u003c/li\u003e\n\u003cli\u003eExamples\u003c/li\u003e\n\u003cli\u003ePlugins\u003c/li\u003e\n\u003cli\u003eTools\u003c/li\u003e\n\u003cli\u003eNews\u003c/li\u003e\n\u003ch1\u003e\n    Datasette\n\u003c/h1\u003e\nFind stories in data\n```\nAll attributes will be removed from the tags, except for the `id=` and `class=` attribute since those may provide further useful hints to the language model.\n\nThe `href` attribute on links, the `alt` attribute on images and the `name` and `value` attributes on `meta` tags are kept as well.\n\nYou can also specify a bundle of tags. For example, `strip-tags -t hs` will keep the tag markup for all levels of headings.\n\nThe following bundles can be used:\n\n\u003c!-- [[[cog\nimport cog\nfrom strip_tags.lib import BUNDLES\nlines = []\nfor name, tags in BUNDLES.items():\n    lines.append(\"- `-t {}`: {}\".format(name, \", \".join(\"`\u003c{}\u003e`\".format(tag) for tag in tags)))\ncog.out(\"\\n\".join(lines))\n]]] --\u003e\n- `-t hs`: `\u003ch1\u003e`, `\u003ch2\u003e`, `\u003ch3\u003e`, `\u003ch4\u003e`, `\u003ch5\u003e`, `\u003ch6\u003e`\n- `-t metadata`: `\u003ctitle\u003e`, `\u003cmeta\u003e`\n- `-t structure`: `\u003cheader\u003e`, `\u003cnav\u003e`, `\u003cmain\u003e`, `\u003carticle\u003e`, `\u003csection\u003e`, `\u003caside\u003e`, `\u003cfooter\u003e`\n- `-t tables`: `\u003ctable\u003e`, `\u003ctr\u003e`, `\u003ctd\u003e`, `\u003cth\u003e`, `\u003cthead\u003e`, `\u003ctbody\u003e`, `\u003ctfoot\u003e`, `\u003ccaption\u003e`, `\u003ccolgroup\u003e`, `\u003ccol\u003e`\n- `-t lists`: `\u003cul\u003e`, `\u003col\u003e`, `\u003cli\u003e`, `\u003cdl\u003e`, `\u003cdd\u003e`, `\u003cdt\u003e`\n\u003c!-- [[[end]]] --\u003e\n\n## As a Python library\n\nYou can use `strip-tags` from Python code too. The function signature looks like this:\n\n\u003c!-- [[[cog\nimport ast\nmodule = ast.parse(open(\"strip_tags/lib.py\").read())\nstrip_tags = [\n    fn for fn in module.body\n    if getattr(fn, 'name', None) == 'strip_tags'\n][0]\ncode = ast.unparse(strip_tags)\ndefline = code.split(\"\\n\")[0]\ncode = (\n    ',\\n    '.join(defline.split(', ')).replace(\") -\u003e\", \"\\n) -\u003e\").replace(\"strip_tags(\", \"strip_tags(\\n    \")\n)\ncog.out(\"```python\\n{}\\n```\".format(code))\n]]] --\u003e\n```python\ndef strip_tags(\n    input: str,\n    selectors: Optional[Iterable[str]]=None,\n    *,\n    removes: Optional[Iterable[str]]=None,\n    minify: bool=False,\n    remove_blank_lines: bool=False,\n    first: bool=False,\n    keep_tags: Optional[Iterable[str]]=None,\n    all_attrs: bool=False\n) -\u003e str:\n```\n\u003c!-- [[[end]]] --\u003e\n\nHere's an example:\n```python\nfrom strip_tags import strip_tags\n\nhtml = \"\"\"\n\u003cdiv\u003e\n\u003ch1\u003eThis has tags\u003c/h1\u003e\n\n\u003cp\u003eAnd whitespace too\u003c/p\u003e\n\u003c/div\u003e\nIgnore this bit.\n\"\"\"\nstripped = strip_tags(html, [\"div\"], minify=True, keep_tags=[\"h1\"])\nprint(stripped)\n```\nOutput:\n```\n\u003ch1\u003eThis has tags\u003c/h1\u003e\n\nAnd whitespace too\n```\nUse `remove_blank_lines=True` to remove any remaining blank lines from the output.\n\n## strip-tags --help\n\n\u003c!-- [[[cog\nimport cog\nfrom strip_tags import cli\nfrom click.testing import CliRunner\nrunner = CliRunner()\nresult = runner.invoke(cli.cli, [\"--help\"])\nhelp = result.output.replace(\"Usage: cli\", \"Usage: strip-tags\")\ncog.out(\n    \"```\\n{}\\n```\".format(help)\n)\n]]] --\u003e\n```\nUsage: strip-tags [OPTIONS] [SELECTORS]...\n\n  Strip tags from HTML, optionally from areas identified by CSS selectors\n\n  Example usage:\n\n      cat input.html | strip-tags \u003e output.txt\n\n  To run against just specific areas identified by CSS selectors:\n\n      cat input.html | strip-tags .entry .footer \u003e output.txt\n\nOptions:\n  --version             Show the version and exit.\n  -r, --remove TEXT     Remove content in these selectors\n  -i, --input FILENAME  Input file\n  -m, --minify          Minify whitespace\n  -t, --keep-tag TEXT   Keep these \u003ctags\u003e\n  --all-attrs           Include all attributes on kept tags\n  --first               First element matching the selectors\n  --help                Show this message and exit.\n\n```\n\u003c!-- [[[end]]] --\u003e\n\n## Development\n\nTo contribute to this tool, first checkout the code. Then create a new virtual environment:\n```bash\ncd strip-tags\npython -m venv venv\nsource venv/bin/activate\n```\nNow install the dependencies and test dependencies:\n```bash\npip install -e '.[test]'\n```\nTo run the tests:\n```bash\npytest\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonw%2Fstrip-tags","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsimonw%2Fstrip-tags","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonw%2Fstrip-tags/lists"}