{"id":31640297,"url":"https://github.com/lorien/domselect","last_synced_at":"2025-10-07T02:21:56.508Z","repository":{"id":312500193,"uuid":"1047521865","full_name":"lorien/domselect","owner":"lorien","description":"Universal interface to work with DOM built by different HTML parsing engines.","archived":false,"fork":false,"pushed_at":"2025-09-14T23:09:26.000Z","size":51,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-15T01:08:50.898Z","etag":null,"topics":["css","dom","html","html-parser","html-parsing","lexbor","selectolax","selector","selectors","selectors-api","xpath"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lorien.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-30T15:55:37.000Z","updated_at":"2025-09-14T23:09:29.000Z","dependencies_parsed_at":"2025-08-31T05:43:18.621Z","dependency_job_id":null,"html_url":"https://github.com/lorien/domselect","commit_stats":null,"previous_names":["lorien/domselect"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lorien/domselect","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lorien%2Fdomselect","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lorien%2Fdomselect/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lorien%2Fdomselect/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lorien%2Fdomselect/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lorien","download_url":"https://codeload.github.com/lorien/domselect/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lorien%2Fdomselect/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278708061,"owners_count":26031944,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-07T02:00:06.786Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["css","dom","html","html-parser","html-parsing","lexbor","selectolax","selector","selectors","selectors-api","xpath"],"created_at":"2025-10-07T02:21:51.092Z","updated_at":"2025-10-07T02:21:56.503Z","avatar_url":"https://github.com/lorien.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Domselect\n\nDomselect provides univeral interface to work with structure of HTML document built by one of supported HTML\nprocessing engines. To work with HTML document you have to create so-called selector object from raw content of HTML document.\nThat selector will be bound to the root node of HTML structure. Then you can call different methods of these selector\nto build other selectors bound to nested parts of HTML structure.\n\nSelector object extracts low-level nodes from DOM constructed by HTML processing backend and wraps them\ninto high-level selector interface. If you need, you can always access low-level node stored in selector object.\n\n### Selector Backends\n\nDomselect library provides these selectors:\n\n1. LexborSelector powered by [selectolax](https://github.com/rushter/selectolax)\n    and [lexbor](https://github.com/lexbor/lexbor) libraries. The type of raw node is `selectolax.lexbor.LexborNode`.\n    Query language is CSS. Lexbor parser is x3-x4 times faster than lxml parser.\n\n2. LxmlCssSelector powered by [lxml](https://github.com/lxml/lxml) library. The type of raw node is `lxml.html.HtmlElement`.\n    Query language is CSS.\n\n2. LxmlXpathSelector powered by [lxml](https://github.com/lxml/lxml) library. The type of raw node is `lxml.html.HtmlElement`.\n    Query language is XPATH.\n\n### Selector Creating\n\nTo create lexbor selector from content of HTML document:\n\n```python\nfrom domselect import LexborSelector\nsel = LexborSelector.from_content(\"\u003cdiv\u003etest\u003c/div\u003e\")\n```\n\nAlso you can create selector from raw node:\n\n```python\nfrom domselect import LexborSelector\nfrom selectolax.lexbor import LexborHTMLParser\nnode = LexborHTMLParser(\"\u003cdiv\u003etest\u003c/div\u003e\").css_first(\"div\")\nsel = LexborSelector(node)\n```\n\nSame goes for lxml backend. Here is an example of creating lxml selector from raw node:\n\n```python\nfrom lxml.html import fromstring\nfrom domselect import LxmlCssSelector, LxmlXpathSelector\nnode = fromstring(\"\u003cdiv\u003etest\u003c/div\u003e\")\nsel = LxmlCssSelector(node)\n# or\nsel = LxmlXpathSelector(node)\n```\n\n### Node Traversal Methods\n\nEach of these methods return other selectors of same type i.e. LexborSelector return\nother LexborSelectors and LxmlCssSelector returns other LxmlCssSelectors.\n\nMethod `find(query: str)` returns list of selectors bound to raw nodes found by query.\n\nMethod `first(query: str)` returns `None` of selector bound to first raw node found by query.\n\nThere is similar `find_raw` and `first_raw` methods which works in same way but returns low-level raw nodes\ni.e. they do not wrap found nodes into selector interface.\n\nMethod `parent()` returns selector bound to raw node which is parent to raw node of current selector.\n\nMethod `exists(query: str)` returns boolean flag indicates if any node has been found by query.\n\nMethod `first_contains(query: str, pattern: str[, default: None])` returns selector bound to first raw node\nfound by query and which contains text as `pattern` parameter. If node is not found then\n`NodeNotFoundError` is raised. You can pass `default=None` optional parameter to return `None` in case\nof node is not found.\n\n\n### Node Properties Methods\n\nMethod `attr(name: str[, default: None|str])` returns content of node's attribute of given name.\nIf node does not have such attribute the `AttributeNotFoundError` is raised. If you pass optional\n`default: None|str` parameter the method will return `None` or `str` if attribute does not exists.\n\nMethod `text([strip: bool])` returns text content of current node and all its sub-nodes. By default\nreturned text is stripped at beginning and ending from whitespaces, tabulations and line-breaks. You\ncan turn off striping by passing `strip=False` parameter.\n\nMethod `tag()` returns tag name of raw node to which current selector is bound.\n\n### Traversal and Properties Methods\n\nThese methods combine two operations: search node by query and do something on found node. They are helful\nif you want to get text or attribute from found node, but this node might not exist. Such methods allows you\nto return reasonable default value in case node is not found. On contrary, if you use call chain like `first().text()`\nthen you'll not be able to return default value from `text()` call because `first()` will raise Exception if\nnode is not found.\n\nMethod `first_attr(query: str, name: str[, default: None|str])` returns content of attribute of given name of node\nfound by given query.  If node does not have such attribute the `AttributeNotFoundError` is raised.\nIf node is not found by given query the `NodeNotFoundError` is raised. If you pass optional\n`default: None|str` parameter the method will return `None` or `str` instead of rasing exceptions.\n\nMethod `first_text(query: str[, default: None|str, strip: bool])` returns text content of raw node (and all its\nsub-nodes) found by given query. If node is not found the `NodeNotFoundError` is raised. Use optional `default: None|str`\nparametere to return `None` or `str` instead of raising exceptions. You can control text stripping with `strip`\nparameter (see description of `text()` method).\n\n### Usage example\n\nThis code downloads telegram channel preview page and parse external links from it.\n\n```python\nfrom html import unescape\nfrom urllib.request import urlopen\n\nfrom domselect import LexborSelector\n\n\ncontent = urlopen(\"https://t.me/s/centralbank_russia\").read()\nsel = LexborSelector.from_content(content)\nfor msg_node in sel.find(\".tgme_widget_message_wrap\"):\n    msg_date = msg_node.first_attr(\n        \".tgme_widget_message_date time\", \"datetime\"\n    )\n    for text_node in msg_node.find(\".tgme_widget_message_text\"):\n        print(\"Message by {}\".format(msg_date))\n        for link_node in text_node.find(\"a[href]\"):\n            url = link_node.attr(\"href\")\n            if url.startswith(\"http\"):\n                print(\" - {}\".format(unescape(url)))\n```\n\nIf you prefer XPATH, here is same task implemented with LxmlXpathSelector:\n\n```python\nfrom html import unescape\nfrom urllib.request import urlopen\n\nfrom domselect import LxmlXpathSelector\n\n\ncontent = urlopen(\"https://t.me/s/centralbank_russia\").read()\nsel = LxmlXpathSelector.from_content(content)\nfor msg_node in sel.find('//*[contains(@class, \"tgme_widget_message_wrap\")]'):\n    msg_date = msg_node.first_attr(\n        './/*[contains(@class, \"tgme_widget_message_date\")]/time', \"datetime\"\n    )\n    for text_node in msg_node.find(\n        './/*[contains(@class, \"tgme_widget_message_text\")]'\n    ):\n        print(\"Message by {}\".format(msg_date))\n        for link_node in text_node.find(\".//a[@href]\"):\n            url = link_node.attr(\"href\")\n            if url.startswith(\"http\"):\n                print(\" - {}\".format(unescape(url)))\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Florien%2Fdomselect","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Florien%2Fdomselect","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Florien%2Fdomselect/lists"}