{"id":13490507,"url":"https://github.com/khromov/sitemap-cache-warmer","last_synced_at":"2025-03-17T08:37:23.438Z","repository":{"id":24898924,"uuid":"28315378","full_name":"khromov/sitemap-cache-warmer","owner":"khromov","description":"Visits pages based on a sitemap to keep your cache warm","archived":false,"fork":false,"pushed_at":"2020-11-15T12:56:00.000Z","size":19,"stargazers_count":103,"open_issues_count":5,"forks_count":28,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-03-14T20:58:07.077Z","etag":null,"topics":["hacktoberfest"],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/khromov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-12-21T22:22:26.000Z","updated_at":"2025-02-12T02:31:49.000Z","dependencies_parsed_at":"2022-07-27T05:02:03.061Z","dependency_job_id":null,"html_url":"https://github.com/khromov/sitemap-cache-warmer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/khromov%2Fsitemap-cache-warmer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/khromov%2Fsitemap-cache-warmer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/khromov%2Fsitemap-cache-warmer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/khromov%2Fsitemap-cache-warmer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/khromov","download_url":"https://codeload.github.com/khromov/sitemap-cache-warmer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243852499,"owners_count":20358271,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hacktoberfest"],"created_at":"2024-07-31T19:00:47.740Z","updated_at":"2025-03-17T08:37:23.161Z","avatar_url":"https://github.com/khromov.png","language":"PHP","funding_links":[],"categories":["PHP"],"sub_categories":[],"readme":"Sitemap Cache Warmer\n====================\n\nThis PHP script crawls URL:s based on a sitemap. It is used to keep your cache warm by visiting all the pages in your sitemap\nat regular intervals. It supports sub-sitemap (Sitemap index).\n\n#### Usage\n\nRename config.php.example to config.php and change the key parameter to a secret value. Upload the file onto your web host, preferably into\nits own folder (for example, /warm-cache)\n\nOnce you have uploaded this file onto your web host, you can visit the following URL to traverse a sitemap and visit all its URL:s:\n\n```\nhttp://example.com/warm-cache/warm.php?key=SECRET_KEY\u0026url=http://example.com/sitemap.xml\u0026sleep=0\n```\n\n###### Available parameters\n\n**key** - Secret key, as entered in config.php (Required)\n**url** - URL to the root sitemap, usually /sitemap.xml (Required)\n**sleep** - Amount of time to sleep between each request in seconds. Used for throttling on slow hosts. (Optional, default is to not throttle.)\n**from** - Number of the url to start with. (Optional, default is 0).\n**to** - Number of the url to stop. Useful to test some URLs on a heavy sitemap (Optional, default is till the end of the sitemap)\n\n#### Scheduling the crawl\n\nYou will need to use CRON to schedule the crawls as often as you wish. Here's an example using cURL and crontab to crawl once every hour:\n\n```\n0 * * * * curl \"http://example.com/warm-cache/warm.php?key=SECRET_KEY\u0026url=http://example.com/sitemap.xml\u0026sleep=0\" \u003e/dev/null 2\u003e\u00261\n```\n\nIf your host provides a CRON URL visiting function, all you need to do is enter the URL, as described in the \"Usage\" section.\n\n#### Output\n\nThe script will provide a JSON output with stats about the crawl, example:\n\n```json\n{\n    \"status\": \"OK\",\n    \"message\": \"Processed sitemap: http://example.com/sitemap.xml\",\n    \"count\" : 4,\n    \"duration\": 9.5575199127197,\n    \"log\": [\n        \"Processed sub-sitemap: http://example.com/post-sitemap.xml\",\n        \"Processed sub-sitemap: http://example.com/page-sitemap.xml\",\n    ],\n    \"visited_urls\": [\n        \"http://example.com/page1/\",\n        \"http://example.com/page2/\",\n        \"http://example.com/page3/\",\n        \"http://example.com/page4/\",\n    ]\n}\n```\n\n#### Reporting unaccessible pages\nYou can set up mail alert when some URLs cannot be accessed. Just modify your `config.php` like this:\n```php\n\u003c?php\nreturn array(\n    'key' =\u003e '9f316c95a356aab49cf5e4fcf3418295' // Secret key to allow traversing sitemaps\n    'reportProblematicUrls' =\u003e true,\n    'reportProblematicUrlsTo' =\u003e \"your-address@example.com\"\n);\n```\n\nURL is reported whencannot be opened with `file_get_contents()`. Proper handling of status codes will be added soon.\n\n#### Using the CLI\n\nAlso you can launch the script from the CLI to bypass the common errors of timeout (504) from an Nginx server.\n\n```\nphp /whatever/you/have/uploaded/it/warm.php url=http://example.com/sitemap.xml sleep=0 key=SECRET_KEY\nphp /whatever/you/have/uploaded/it/warm.php url=http://example.com/sitemap.xml sleep=0 key=SECRET_KEY from=10 to=100\nphp /whatever/you/have/uploaded/it/warm.php url=http://example.com/sitemap.xml sleep=0 key=SECRET_KEY to=25\n```\n\n#### Crawl strategies\n\nIf you employ time-based static page cache, you can schedule your crawls to coincide with half the cache expiration time.\n\nFor example, if your expiration time is one hour (3600 seconds), you can schedule the crawls to take place every thirty minutes (1800 seconds).\n\nIf you have a lot of pages and few visitors, this may cause increased load on the server. For low-traffic deployments, use a long cache expiration time (24 hours or more) and invalidate cache when page content changes.\n\n#### Requirements\n\n* SimpleXML\n* allow_url_fopen in php.ini (Enabled on most hosts)\n\n#### Compatibility\n\nThe plugin has been tested with the WordPress plugins [Yoast WordPress SEO](https://wordpress.org/plugins/wordpress-seo/) and [Google XML Sitemaps](https://wordpress.org/plugins/google-sitemap-generator/). It should work with any sitemap which conforms to the [sitemap standard](http://www.sitemaps.org/protocol.html).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkhromov%2Fsitemap-cache-warmer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkhromov%2Fsitemap-cache-warmer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkhromov%2Fsitemap-cache-warmer/lists"}