Test Title
Test Paragraph 1
Test Paragraph 2
https://github.com/ixnode/php-web-crawler
This PHP class allows you to crawl recursively a given html page (or a given html file) and collect some data from it.
https://github.com/ixnode/php-web-crawler
Last synced: 8 months ago
JSON representation
This PHP class allows you to crawl recursively a given html page (or a given html file) and collect some data from it.
- Host: GitHub
- URL: https://github.com/ixnode/php-web-crawler
- Owner: ixnode
- License: mit
- Created: 2024-02-24T14:07:40.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-02-28T01:18:31.000Z (over 2 years ago)
- Last Synced: 2024-12-27T01:41:38.555Z (over 1 year ago)
- Language: PHP
- Size: 204 KB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE.md
Awesome Lists containing this project
README
# PHP Web Crawler
[](https://github.com/ixnode/php-web-crawler/releases)
[](https://github.com/ixnode/php-web-crawler/releases)
[](https://www.php.net/supported-versions.php)
[](https://phpstan.org/user-guide/rule-levels)
[](https://phpunit.de)
[](https://www.php-fig.org/psr/psr-12/)
[](https://github.com/phpmd/phpmd)
[](https://github.com/rectorphp/rector)
[](https://github.com/ixnode/php-api-version-bundle/blob/master/LICENSE)
> This PHP class allows you to crawl recursively a given html page (or a given html file) and collect some data from it.
> Simply define the url (or a html file) and a set of xpath expressions which should map with the output data object.
> The final representation will be a php array which can be easily converted into the json format for further
> processing.
## 1. Installation
```shell
composer require ixnode/php-web-crawler
```
```shell
vendor/bin/php-web-crawler -V
```
```shell
php-web-crawler 0.1.0 (02-24-2024 14:46:26) - Björn Hempel
```
## 2. Usage
### 2.1 PHP Code
```php
use Ixnode\PhpWebCrawler\Output\Field;
use Ixnode\PhpWebCrawler\Source\Raw;
use Ixnode\PhpWebCrawler\Value\Text;
use Ixnode\PhpWebCrawler\Value\XpathTextNode;
$rawHtml = <<
Test Page
Test Title
Test Paragraph
HTML;
$html = new Raw(
$rawHtml,
new Field('version', new Text('1.0.0')),
new Field('title', new XpathTextNode('//h1')),
new Field('paragraph', new XpathTextNode('//p'))
);
$html->parse()->getJsonStringFormatted();
// See below
```
### 2.2 JSON result
```json
{
"version": "1.0.0",
"title": "Test Title",
"paragraph": "Test Paragraph"
}
```
## 3. Advanced usage
### 3.1 Group
#### PHP Code
```php
use Ixnode\PhpWebCrawler\Output\Field;
use Ixnode\PhpWebCrawler\Output\Group;
use Ixnode\PhpWebCrawler\Source\Raw;
use Ixnode\PhpWebCrawler\Value\XpathTextNode;
$rawHtml = <<
Test Page
Test Title
Test Paragraph 1
Test Paragraph 2
HTML;
$html = new Raw(
$rawHtml,
new Field('title', new XpathTextNode('/html/head/title')),
new Group(
'content',
new Group(
'header',
new Field('h1', new XpathTextNode('/html/body//h1')),
),
new Group(
'text',
new Field('p1', new XpathTextNode('/html/body//p[@class="paragraph-1"]')),
new Field('p2', new XpathTextNode('/html/body//p[@class="paragraph-2"]')),
)
)
);
$html->parse()->getJsonStringFormatted();
// See below
```
#### JSON result
```json
{
"title": "Test Page",
"content": {
"header": {
"h1": "Test Title"
},
"text": {
"p1": "Test Paragraph 1",
"p2": "Test Paragraph 2"
}
}
}
```
### 3.2 XpathSection
#### PHP Code
```php
use Ixnode\PhpWebCrawler\Output\Field;
use Ixnode\PhpWebCrawler\Output\Group;
use Ixnode\PhpWebCrawler\Source\Raw;
use Ixnode\PhpWebCrawler\Source\XpathSection;
use Ixnode\PhpWebCrawler\Value\XpathTextNode;
$rawHtml = <<
Test Page
HTML;
$html = new Raw(
$rawHtml,
new Field('title', new XpathTextNode('/html/head/title')),
new Group(
'content',
new XpathSection(
'/html/body//div[@class="content"]',
new Group(
'header',
new Field('h1', new XpathTextNode('./h1')),
),
new Group(
'text',
new Field('p1', new XpathTextNode('./p[@class="paragraph-1"]')),
new Field('p2', new XpathTextNode('./p[@class="paragraph-2"]')),
)
)
)
);
$html->parse()->getJsonStringFormatted();
// See below
```
#### JSON result
```json
{
"title": "Test Page",
"content": {
"header": {
"h1": "Test Title"
},
"text": {
"p1": "Test Paragraph 1",
"p2": "Test Paragraph 2"
}
}
}
```
### 3.3 XpathSection (flat)
#### PHP Code
```php
use Ixnode\PhpWebCrawler\Output\Field;
use Ixnode\PhpWebCrawler\Output\Group;
use Ixnode\PhpWebCrawler\Source\Raw;
use Ixnode\PhpWebCrawler\Source\XpathSections;
use Ixnode\PhpWebCrawler\Value\XpathTextNode;
$rawHtml = <<
Test Page
Test Title
Test Paragraph 1
Test Paragraph 2
- Test Item 1
- Test Item 2
HTML;
$html = new Raw(
$rawHtml,
new Field('title', new XpathTextNode('/html/head/title')),
new Group(
'hits',
new XpathSections(
'/html/body//div[@class="content"]/ul',
new XpathTextNode('./li/text()'),
)
)
);
$html->parse()->getJsonStringFormatted();
// See below
```
#### JSON result
```json
{
"title": "Test Page",
"hits": [
[
"Test Item 1",
"Test Item 2"
]
]
}
```
### 3.3 XpathSection (structured)
#### PHP Code
```php
use Ixnode\PhpWebCrawler\Output\Field;
use Ixnode\PhpWebCrawler\Output\Group;
use Ixnode\PhpWebCrawler\Source\Raw;
use Ixnode\PhpWebCrawler\Source\XpathSections;
use Ixnode\PhpWebCrawler\Value\XpathTextNode;
$rawHtml = <<
Test Page
Test Title
Test Paragraph 1
Test Paragraph 2
Caption 1
Cell 1
Caption 2
Cell 2
HTML;
$html = new Raw(
$rawHtml,
new Field('title', new XpathTextNode('/html/head/title')),
new Group(
'hits',
new XpathSections(
'/html/body//div[@class="content"]/table/tbody/tr',
new Field('caption', new XpathTextNode('./th/text()')),
new Field('content', new XpathTextNode('./td/text()')),
)
)
);
$html->parse()->getJsonStringFormatted();
// See below
```
#### JSON result
```json
{
"title": "Test Page",
"hits": [
{
"caption": "Caption 1",
"content": "Cell 1"
},
{
"caption": "Caption 2",
"content": "Cell 2"
}
]
}
```
## 4. More examples
* [examples/converter.php](examples/converter.php)
* [examples/group.php](examples/group.php)
* [examples/section.php](examples/section.php)
* [examples/sections-recursive-url.php](examples/sections-recursive-url.php)
* [examples/sections.php](examples/sections.php)
* [examples/simple-wiki-page.php](examples/simple-wiki-page.php)
## 5. Development
```bash
git clone git@github.com:ixnode/php-web-crawler.git && cd php-web-crawler
```
```bash
composer install
```
```bash
composer test
```
## 6. License
This library is licensed under the MIT License - see the [LICENSE.md](/LICENSE.md) file for details.