{"id":16843392,"url":"https://github.com/barttc/fetch-sitemap","last_synced_at":"2025-08-01T06:41:45.242Z","repository":{"id":65398676,"uuid":"591656010","full_name":"bartTC/fetch-sitemap","owner":"bartTC","description":null,"archived":false,"fork":false,"pushed_at":"2024-10-17T19:57:48.000Z","size":680,"stargazers_count":4,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-30T05:05:14.713Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bartTC.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-21T12:42:40.000Z","updated_at":"2024-10-17T19:57:51.000Z","dependencies_parsed_at":"2024-06-15T10:48:22.676Z","dependency_job_id":"2660e8e7-dbed-4f68-8b5f-24a72ead86f2","html_url":"https://github.com/bartTC/fetch-sitemap","commit_stats":{"total_commits":10,"total_committers":1,"mean_commits":10.0,"dds":0.0,"last_synced_commit":"26ee84bcb1b3c14a50684a4f2685f3d7290b167a"},"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"purl":"pkg:github/bartTC/fetch-sitemap","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bartTC%2Ffetch-sitemap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bartTC%2Ffetch-sitemap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bartTC%2Ffetch-sitemap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bartTC%2Ffetch-sitemap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bartTC","download_url":"https://codeload.github.com/bartTC/fetch-sitemap/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bartTC%2Ffetch-sitemap/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262714504,"owners_count":23352464,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T12:50:51.501Z","updated_at":"2025-06-30T05:05:16.113Z","avatar_url":"https://github.com/bartTC.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fetch-sitemap\n\nRetrieves all URLs of a given sitemap.xml URL and fetches each page one by one. \nUseful for (load) testing the entire site for error responses.\n\n![Sample Output](https://raw.githubusercontent.com/bartTC/fetch-sitemap/main/example.png)\n\n## Installation\n\n```bash \n$ pip install fetch-sitemap\n```\n\n## Usage \n\n```\n$ fetch-sitemap --help\n\n Usage: fetch-sitemap [OPTIONS] SITEMAP_URL\n\n Fetch a given sitemap and retrieve all URLs in it.\n\n╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮\n│ --basic-auth -a TEXT Basic auth information. Format: 'username:password' │\n│ --limit -l INT [\u003e=1] Maximum number of URLs to fetch from the given sitemap.xml. │\n│ --recursive/--no-recursive Recursively fetch all sitemap documents from the given sitemap.xml. [default: recursive] │\n│ --concurrency-limit -c INT [\u003e=1] Max number of concurrent requests. [default: 5; \u003e=1] │\n│ --request-timeout -t INT [\u003e=1] Timeout for fetching a URL in seconds. [default: 30; \u003e=1] │\n│ --random -r Append a random string like ?12334232343 to each URL to bypass frontend cache. │\n│ --random-length INT [1 to 100] Length of the --random hash. [default: 15; 1 to 100] │\n│ --report-path -p FILE Store results in a CSV file. Example: ./report.csv │\n│ --output-dir -o DIRECTORY Store all fetched sitemap documents in this folder. Example: /tmp/my.domain.com/ │\n│ --slow-threshold FLOAT [\u003e=0.0] Responses slower than this (in seconds) are considered 'slow'. [default: 5.0; \u003e=0.0] │\n│ --slow-num INTEGER OR \"ALL\" How many 'slow' responses to show. [default: 10] │\n│ --user-agent TEXT User-Agent string set in the HTTP header. [default: Mozilla/5.0 (compatible; fetch-sitemap/23)] │\n│ --version Show the version and exit. │\n│ --help Show this message and exit. │\n╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\n```\n\n## 🤺 Local Development\n\n```bash\npoetry install\npoetry run fetch-sitemap -h\npoetry run ./tests.sh\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbarttc%2Ffetch-sitemap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbarttc%2Ffetch-sitemap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbarttc%2Ffetch-sitemap/lists"}