https://github.com/dr4g0nsr/sitemap-crawler
Crawler using sitemap to crawl site/regenerate cache
https://github.com/dr4g0nsr/sitemap-crawler
crawler php
Last synced: 5 months ago
JSON representation
Crawler using sitemap to crawl site/regenerate cache
- Host: GitHub
- URL: https://github.com/dr4g0nsr/sitemap-crawler
- Owner: dr4g0nsr
- License: lgpl-2.1
- Created: 2022-10-09T19:11:18.000Z (over 3 years ago)
- Default Branch: master
- Last Pushed: 2022-11-11T21:17:11.000Z (over 3 years ago)
- Last Synced: 2023-05-23T18:52:39.438Z (about 3 years ago)
- Topics: crawler, php
- Language: PHP
- Homepage:
- Size: 48.8 KB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# ALPHA VERSION, DO NOT USE ON PRODUCTION
[](https://github.com/dr4g0nsr/sitemap-crawler/actions)
## Sitemap Crawler
Crawler using sitemap to crawl site/regenerate cache.
Files are not stored, point is just to trigger url.
## Get code using composer
```
composer require dr4g0nsr/sitemap-crawler
```
## How to implement
Create config.php:
```
0,
"excluded" => []
];
```
Use code like this:
```
0, 'verbose' => true]);
$crawler->loadConfig(__DIR__ . '/config.php');
$sitemap = $crawler->getSitemap($url);
$crawler->crawlURLS($sitemap);
```
That would be simplest code, you can also find it in test subdir under vendor/dr4g0nsr/SitemapCrawler/test.