{"id":22515492,"url":"https://github.com/kylesmith19091/beautiful-soup-cpp","last_synced_at":"2025-06-20T17:36:17.482Z","repository":{"id":124213589,"uuid":"448076866","full_name":"KyleSmith19091/Beautiful-Soup-CPP","owner":"KyleSmith19091","description":"Beautiful soup inspired program for extracting data from HTML documents.","archived":false,"fork":false,"pushed_at":"2022-01-15T09:59:47.000Z","size":575,"stargazers_count":1,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-28T02:21:17.540Z","etag":null,"topics":["beautifulsoup","cpp11","parser","treesitter-api"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KyleSmith19091.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-14T18:58:37.000Z","updated_at":"2022-01-14T19:00:01.000Z","dependencies_parsed_at":null,"dependency_job_id":"550bd3d4-d583-4626-bda6-286f71b7b35c","html_url":"https://github.com/KyleSmith19091/Beautiful-Soup-CPP","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/KyleSmith19091/Beautiful-Soup-CPP","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KyleSmith19091%2FBeautiful-Soup-CPP","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KyleSmith19091%2FBeautiful-Soup-CPP/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KyleSmith19091%2FBeautiful-Soup-CPP/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KyleSmith19091%2FBeautiful-Soup-CPP/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KyleSmith19091","download_url":"https://codeload.github.com/KyleSmith19091/Beautiful-Soup-CPP/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KyleSmith19091%2FBeautiful-Soup-CPP/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260988609,"owners_count":23093528,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beautifulsoup","cpp11","parser","treesitter-api"],"created_at":"2024-12-07T03:32:49.738Z","updated_at":"2025-06-20T17:36:12.466Z","avatar_url":"https://github.com/KyleSmith19091.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ⚡️ HTML Document Data Extractor\n\nThis project was inspired by the BeautifulSoup python library and was used as an opportunity to learn the tree sitter parser api. \n\n## Project Components\n\nThe project can be broken into two main components:\n1. Tree-sitter API\n2. CPP Interface\n\nThe Tree-sitter api does did not have a C++ language binding as it is written in C so I wrote a C++ layer on top of this api. The Tree-sitter library does not have native HTML parsing capabilities as\nit is meant to be parser generator tool, but does provide a query language interface which will be discussed later. This project uses a HTML grammar which can be found here: \u003ca href=\"https://github.com/tree-sitter/tree-sitter-html\"\u003eTree-sitter-html\u003c/a\u003e\n\n## Usage\nThe BeautifulSoup object is the only interface that is required to use the library.\n\n```c++\nBeautifulSoup soup(\"url\");\n```\nUpon creating the BeautifulSoup object given a valid URL it will perform a cURL request using the cURL library and parse the html document at that location. We now have an AST(Abstract Syntax Tree) which we can now explore.\n\nThe BeautifulSoup object provides two methods:\n1. find_all\n2. find\n\n### Find_all\nThe find_all can be used to find all occurences of a specific tag in the html document. This is done by first extracting all instances of tags with the same 'tag name' as given in the parameter of the function call. We do this by performing\na query on the AST which Tree-sitter allows us to do through its very own query language which is very 'scheme' like.\n\nThe query for finding the tag names loooks something like this:\n```scheme\n(start_tag (tag_name) @name)\n```\n\n#### std::vector\\\u003cTagNode\u003e BeautifulSoup::find_all(const std::string\u0026 tagName)\n\n```c++\nBeautifulSoup soup(\"url\");\nauto docSpans = soup.find_all(\"span\");\n```\nThe above example will extract all nodes from the AST, starting from the root node, with the span tag name. We can further filter the search results(See next example).\n\n#### std::vector\\\u003cTagNode\u003e BeautifulSoup::find_all(const std::string\u0026 tagName, std::map\\\u003cstd::string,std::string\\\u003e attrs)\n```c++\nBeautifulSoup soup(\"url\");\nauto docSpans = soup.find_all(\"span\", {{\"class\", \"social\"}});\n```\nThe above example will extract all nodes from the AST, starting from the root node, with the span tag name and which have the social class.\n\n#### std::vector\\\u003cTagNode\u003e BeautifulSoup::find_all(const Node\u0026 startNode, const std::string\u0026 tagName, std::map\\\u003cstd::string,std::string\\\u003e attrs)\n```c++\nBeautifulSoup soup(\"url\");\nauto articles = soup.find_all(\"article\", {{\"class\", \"featured\"}, {\"data-value\", \"2\"}});\n\nfor(auto\u0026 article : articles) {\n    auto articleTitle = soup.find(article, \"h1\", {{\"class\", \"title\"}});\n    std::cout \u003c\u003c soup.getNodeText(articleTitle) \u003c\u003c \"\\n\";\n}\n```\nThe above example finds all articles in the html document and then proceeds to print the titles of those articles by finding the first h1 tag within that tag with the title class and using the getNodeText helper function to\nextract text data from the source document.\n\n#### TagNode BeautifulSoup::find(const std::string\u0026 tagName)\n```c++\nBeautifulSoup soup(\"url\");\nauto docSpans = soup.find(\"span\");\n```\nThe above example will extract the first node from the AST, starting from the root node, with the span tag name. We can further filter the search results(See next example).\n\n#### TagNode BeautifulSoup::find(const std::string\u0026 tagName, std::map\\\u003cstd::string,std::string\\\u003e attrs)\n```c++\nBeautifulSoup soup(\"url\");\nauto docSpans = soup.find_all(\"span\", {{\"class\", \"social\"}});\n```\nThe above example will extract the first node from the AST, starting from the root node, with the span tag name and includes the social class.\n\n#### TagNode BeautifulSoup::find(const Node\u0026 startNode, const std::string\u0026 tagName, std::map\\\u003cstd::string,std::string\\\u003e attrs)\n```c++\nBeautifulSoup soup(\"url\");\nauto articles = soup.find_all(\"article\", {{\"class\", \"featured\"}, {\"data-value\", \"2\"}});\n\nfor(auto\u0026 article : articles) {\n    auto articleTitle = soup.find(article, \"h1\", {{\"class\", \"title\"}});\n    std::cout \u003c\u003c soup.getNodeText(articleTitle) \u003c\u003c \"\\n\";\n}\n```\nThe above example finds the first article in the html document and then proceeds to print the title of that title by finding the first h1 tag within that tag with the title class and using the getNodeText helper function to\nextract text data from the source document.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkylesmith19091%2Fbeautiful-soup-cpp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkylesmith19091%2Fbeautiful-soup-cpp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkylesmith19091%2Fbeautiful-soup-cpp/lists"}