{"id":21698658,"url":"https://github.com/theohbrothers/pssitescraper","last_synced_at":"2025-04-12T12:41:32.906Z","repository":{"id":50778488,"uuid":"73207072","full_name":"theohbrothers/PSSiteScraper","owner":"theohbrothers","description":"Cmdlets for scraping a site.","archived":false,"fork":false,"pushed_at":"2025-02-13T11:24:48.000Z","size":86,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-03-26T07:22:00.706Z","etag":null,"topics":["html","powershell","pwsh","scrape","site","sitemap","uri","uri-scheme","url","website"],"latest_commit_sha":null,"homepage":"","language":"PowerShell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/theohbrothers.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-11-08T16:50:16.000Z","updated_at":"2025-02-13T11:24:52.000Z","dependencies_parsed_at":"2022-09-12T02:50:43.465Z","dependency_job_id":null,"html_url":"https://github.com/theohbrothers/PSSiteScraper","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theohbrothers%2FPSSiteScraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theohbrothers%2FPSSiteScraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theohbrothers%2FPSSiteScraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theohbrothers%2FPSSiteScraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/theohbrothers","download_url":"https://codeload.github.com/theohbrothers/PSSiteScraper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248568579,"owners_count":21126052,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["html","powershell","pwsh","scrape","site","sitemap","uri","uri-scheme","url","website"],"created_at":"2024-11-25T19:35:53.212Z","updated_at":"2025-04-12T12:41:32.883Z","avatar_url":"https://github.com/theohbrothers.png","language":"PowerShell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PSSiteScraper\n\n[![github-actions](https://github.com/theohbrothers/PSSiteScraper/workflows/ci-master-pr/badge.svg)](https://github.com/theohbrothers/PSSiteScraper/actions)\n[![github-release](https://img.shields.io/github/v/release/theohbrothers/PSSiteScraper?style=flat-square)](https://github.com/theohbrothers/PSSiteScraper/releases/)\n[![powershell-gallery-release](https://img.shields.io/powershellgallery/v/PSSiteScraper?logo=powershell\u0026logoColor=white\u0026label=PSGallery\u0026labelColor=\u0026style=flat-square)](https://www.powershellgallery.com/packages/PSSiteScraper/)\n\nCmdlets for scraping a site.\n\n## Agenda\n\n- Get a site's sitemaps\n- Get a site's published URLs from sitemaps\n- Get URIs from HTML\n\n## Install\n\nOpen [`powershell`](https://docs.microsoft.com/en-us/powershell/scripting/windows-powershell/install/installing-windows-powershell?view=powershell-5.1) or [`pwsh`](https://github.com/powershell/powershell#-powershell) and type:\n\n```powershell\nInstall-Module -Name PSSiteScraper -Repository PSGallery -Scope CurrentUser -Verbose\n```\n\nIf prompted to trust the repository, hit `Y` and `enter`.\n\n## Usage\n\n```powershell\nImport-Module PSSiteScraper\n\n# Get child sitemaps of a parent sitemap.\nGet-Sitemaps -Uri https://example.com/sitemap.xml\n\n# Get URLs from a sitemap\nGet-SitemapUris -Uri https://example.com/sitemap-child.xml\n\n# Get URIs from all tags' attributes of given HTML\nGet-HtmlUris -Html $html\n# Get URIs from all tags' attributes of given HTML of scheme 'foo'. E.g. URI 'foo://bar/baz'\nGet-HtmlUris -Html $html -UriScheme foo\n# Get URIs from all \u003ca\u003e tag's attributes of given HTML\nGet-HtmlUris -Html $html -Tag a -UriScheme https\n# Get URIs from all \u003cimg\u003e tag's 'srcset' attribute of given HTML\nGet-HtmlUris -Html $html -Tag img -Attribute srcset -UriScheme https\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftheohbrothers%2Fpssitescraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftheohbrothers%2Fpssitescraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftheohbrothers%2Fpssitescraper/lists"}