{"id":36976282,"url":"https://github.com/dr4g0nsr/sitemap-crawler","last_synced_at":"2026-01-13T22:11:47.673Z","repository":{"id":61127640,"uuid":"548534394","full_name":"dr4g0nsr/sitemap-crawler","owner":"dr4g0nsr","description":"Crawler using sitemap to crawl site/regenerate cache","archived":false,"fork":false,"pushed_at":"2022-11-11T21:17:11.000Z","size":50,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2023-05-23T18:52:39.438Z","etag":null,"topics":["crawler","php"],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-2.1","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dr4g0nsr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-10-09T19:11:18.000Z","updated_at":"2022-11-06T21:33:32.000Z","dependencies_parsed_at":"2023-01-21T12:30:37.025Z","dependency_job_id":null,"html_url":"https://github.com/dr4g0nsr/sitemap-crawler","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"purl":"pkg:github/dr4g0nsr/sitemap-crawler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dr4g0nsr%2Fsitemap-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dr4g0nsr%2Fsitemap-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dr4g0nsr%2Fsitemap-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dr4g0nsr%2Fsitemap-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dr4g0nsr","download_url":"https://codeload.github.com/dr4g0nsr/sitemap-crawler/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dr4g0nsr%2Fsitemap-crawler/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28402192,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-13T14:36:09.778Z","status":"ssl_error","status_checked_at":"2026-01-13T14:35:19.697Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","php"],"created_at":"2026-01-13T22:11:46.937Z","updated_at":"2026-01-13T22:11:47.666Z","avatar_url":"https://github.com/dr4g0nsr.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ALPHA VERSION, DO NOT USE ON PRODUCTION\n\n[![Test status](https://github.com/dr4g0nsr/sitemap-crawler/workflows/Composer/badge.svg)](https://github.com/dr4g0nsr/sitemap-crawler/actions)\n\n## Sitemap Crawler\n\nCrawler using sitemap to crawl site/regenerate cache.\n\nFiles are not stored, point is just to trigger url.\n\n## Get code using composer\n\n```\ncomposer require dr4g0nsr/sitemap-crawler\n```\n\n## How to implement\n\nCreate config.php:\n\n```\n\u003c?php\n\n$settings = [\n    \"sleep\" =\u003e 0,\n    \"excluded\" =\u003e []\n];\n```\n\nUse code like this:\n\n```\n\u003c?php\n\nrequire __DIR__ . '/vendor/autoload.php';\nrequire __DIR__ . '/config.php';\n\nuse dr4g0nsr\\Crawler;\n\n$url = 'https://candymapper.com';\nprint \"Crawler version: \" . Crawler::version() . PHP_EOL;\n\n$crawler = new Crawler(['sleep' =\u003e 0, 'verbose' =\u003e true]);\n$crawler-\u003eloadConfig(__DIR__ . '/config.php');\n$sitemap = $crawler-\u003egetSitemap($url);\n$crawler-\u003ecrawlURLS($sitemap);\n```\n\nThat would be simplest code, you can also find it in test subdir under vendor/dr4g0nsr/SitemapCrawler/test.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdr4g0nsr%2Fsitemap-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdr4g0nsr%2Fsitemap-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdr4g0nsr%2Fsitemap-crawler/lists"}