{"id":18917267,"url":"https://github.com/danielemoraschi/sitemap-common","last_synced_at":"2026-03-11T18:30:19.333Z","repository":{"id":56970287,"uuid":"65993170","full_name":"danielemoraschi/sitemap-common","owner":"danielemoraschi","description":"Simple PHP Sitemap generator and crawler library.","archived":false,"fork":false,"pushed_at":"2016-08-21T23:09:31.000Z","size":56,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-01T01:47:20.867Z","etag":null,"topics":["crawler","php","php-library","php-sitemap-generator","sitemap"],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/danielemoraschi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-08-18T11:53:46.000Z","updated_at":"2017-08-01T22:41:28.000Z","dependencies_parsed_at":"2022-08-21T06:40:23.632Z","dependency_job_id":null,"html_url":"https://github.com/danielemoraschi/sitemap-common","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danielemoraschi%2Fsitemap-common","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danielemoraschi%2Fsitemap-common/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danielemoraschi%2Fsitemap-common/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danielemoraschi%2Fsitemap-common/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/danielemoraschi","download_url":"https://codeload.github.com/danielemoraschi/sitemap-common/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239914929,"owners_count":19717759,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","php","php-library","php-sitemap-generator","sitemap"],"created_at":"2024-11-08T10:25:01.391Z","updated_at":"2026-03-11T18:30:19.299Z","avatar_url":"https://github.com/danielemoraschi.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"# A PHP sitemap generator and crawler\n##\n\n[![Build Status](https://travis-ci.org/danielemoraschi/sitemap-common.png?branch=master)](https://travis-ci.org/danielemoraschi/sitemap-common)\n[![Scrutinizer Quality Score](https://scrutinizer-ci.com/g/danielemoraschi/sitemap-common/badges/quality-score.png?b=master)](https://scrutinizer-ci.com/g/danielemoraschi/sitemap-common/)\n\nThis package provides all of the components to crawl a website and build and write sitemaps file.\n\nExample of console application using the library: [dmoraschi/sitemap-app](https://github.com/danielemoraschi/sitemap-app)\n\n\n## Installation\n\nRun the following command and provide the latest stable version (e.g v1.0.0):\n\n```bash\ncomposer require dmoraschi/sitemap-common\n```\n\nor add the following to your `composer.json` file :\n\n```json\n\"dmoraschi/sitemap-common\": \"1.0.*\"\n``````\n\n`SiteMapGenerator`\n-----\n**Basic usage**\n\n``` php\n$generator = new SiteMapGenerator(\n    new FileWriter($outputFileName),\n    new XmlTemplate()\n);\n```\n\nAdd a URL:\n``` php\n$generator-\u003eaddUrl($url, $frequency, $priority);\n```\n\nAdd a single `SiteMapUrl` object or array:\n``` php\n$siteMapUrl = new SiteMapUrl(\n    new Url($url), $frequency, $priority\n);\n\n$generator-\u003eaddSiteMapUrl($siteMapUrl);\n\n$generator-\u003eaddSiteMapUrls([\n    $siteMapUrl, $siteMapUrl2\n]);\n```\n\nSet the URLs of the sitemap via `SiteMapUrlCollection`:\n``` php\n$siteMapUrl = new SiteMapUrl(\n    new Url($url), $frequency, $priority\n);\n\n$collection = new SiteMapUrlCollection([\n    $siteMapUrl, $siteMapUrl2\n]);\n\n$generator-\u003esetCollection($collection);\n```\n\nGenerate the sitemap:\n``` php\n$generator-\u003eexecute();\n```\n\n`Crawler`\n-----\n**Basic usage**\n\n``` php\n$crawler = new Crawler(\n    new Url($baseUrl),\n    new RegexBasedLinkParser(),\n    new HttpClient()\n);\n```\n\nYou can tell the `Crawler` **not to visit** certain url's by adding policies. Below the default policies provided by the library:\n```php\n$crawler-\u003esetPolicies([\n    'host' =\u003e new SameHostPolicy($baseUrl),\n    'url'  =\u003e new UniqueUrlPolicy(),\n    'ext'  =\u003e new ValidExtensionPolicy(),\n]);\n// or\n$crawler-\u003esetPolicy('host', new SameHostPolicy($baseUrl));\n```\n`SameHostPolicy`, `UniqueUrlPolicy`, `ValidExtensionPolicy` are provided with the library, you can define your own policies by implementing the interface `Policy`.\n\nCalling the function `crawl` the object will start from the base url in the contructor and crawl all the web pages with the specified depth passed as a argument.\nThe function will return with the array of all unique visited `Url`'s:\n```php\n$urls = $crawler-\u003ecrawl($deep);\n```\n\nYou can also instruct the `Crawler` to collect custom data while visiting the web pages by adding `Collector`'s to the main object:\n```php\n$crawler-\u003esetCollectors([\n    'images' =\u003e new ImageCollector()\n]);\n// or\n$crawler-\u003esetCollector('images', new ImageCollector());\n```\nAnd then retrive the collected data:\n```php\n$crawler-\u003ecrawl($deep);\n\n$imageCollector = $crawler-\u003egetCollector('images');\n$data = $imageCollector-\u003egetCollectedData();\n```\n`ImageCollector` is provided by the library, you can define your own collector by implementing the interface `Collector`.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanielemoraschi%2Fsitemap-common","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdanielemoraschi%2Fsitemap-common","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanielemoraschi%2Fsitemap-common/lists"}