{"id":20213856,"url":"https://github.com/dachcom-digital/pimcore-dynamic-search-data-provider-crawler","last_synced_at":"2025-04-10T14:04:52.278Z","repository":{"id":34933430,"uuid":"191616568","full_name":"dachcom-digital/pimcore-dynamic-search-data-provider-crawler","owner":"dachcom-digital","description":"A Spider Crawler Extension for Pimcore Dynamic Search.","archived":false,"fork":false,"pushed_at":"2023-12-14T09:14:54.000Z","size":180,"stargazers_count":8,"open_issues_count":0,"forks_count":7,"subscribers_count":9,"default_branch":"master","last_synced_at":"2024-04-26T13:43:27.579Z","etag":null,"topics":["crawler","index","pimcore","scraper","search"],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dachcom-digital.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-06-12T17:32:13.000Z","updated_at":"2022-10-19T03:35:27.000Z","dependencies_parsed_at":"2023-01-15T10:43:07.070Z","dependency_job_id":"6bc35e65-d690-4b40-8c96-61781c4926d8","html_url":"https://github.com/dachcom-digital/pimcore-dynamic-search-data-provider-crawler","commit_stats":{"total_commits":85,"total_committers":8,"mean_commits":10.625,"dds":0.2823529411764706,"last_synced_commit":"2a4aa67a546d77d76687df815fb7050439a1cd34"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dachcom-digital%2Fpimcore-dynamic-search-data-provider-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dachcom-digital%2Fpimcore-dynamic-search-data-provider-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dachcom-digital%2Fpimcore-dynamic-search-data-provider-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dachcom-digital%2Fpimcore-dynamic-search-data-provider-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dachcom-digital","download_url":"https://codeload.github.com/dachcom-digital/pimcore-dynamic-search-data-provider-crawler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248231430,"owners_count":21069322,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","index","pimcore","scraper","search"],"created_at":"2024-11-14T06:12:23.752Z","updated_at":"2025-04-10T14:04:52.263Z","avatar_url":"https://github.com/dachcom-digital.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dynamic Search | Data Provider: Web Crawler\n\n[![Software License](https://img.shields.io/badge/license-GPLv3-brightgreen.svg?style=flat-square)](LICENSE.md)\n[![Latest Release](https://img.shields.io/packagist/v/dachcom-digital/dynamic-search-data-provider-crawler.svg?style=flat-square)](https://packagist.org/packages/dachcom-digital/dynamic-search-data-provider-crawler)\n[![Tests](https://img.shields.io/github/actions/workflow/status/dachcom-digital/pimcore-dynamic-search-data-provider-crawler/.github/workflows/codeception.yml?branch=master\u0026style=flat-square\u0026logo=github\u0026label=codeception)](https://github.com/dachcom-digital/pimcore-dynamic-search-data-provider-crawler/actions?query=workflow%3ACodeception+branch%3Amaster)\n[![PhpStan](https://img.shields.io/github/actions/workflow/status/dachcom-digital/pimcore-dynamic-search-data-provider-crawler/.github/workflows/php-stan.yml?branch=master\u0026style=flat-square\u0026logo=github\u0026label=phpstan%20level%204)](https://github.com/dachcom-digital/pimcore-dynamic-search-data-provider-crawler/actions?query=workflow%3A\"PHP+Stan\"+branch%3Amaster)\n\nA spider crawler extension for [Pimcore Dynamic Search](https://github.com/dachcom-digital/pimcore-dynamic-search).\n\n\u003e [!CAUTION]  \n\u003e This Connector has reached its end of life and only receives compatibility update. It will not be developed further.\n\u003e Use the [Trinity Data Provider](https://github.com/dachcom-digital/pimcore-dynamic-search-data-provider-trinity) instead!\n\n## Release Plan\n| Release | Supported Pimcore Versions | Supported Symfony Versions | Release Date | Maintained     | Branch                                                                                          |\n|---------|----------------------------|----------------------------|--------------|----------------|-------------------------------------------------------------------------------------------------|\n| **3.x** | `11.0`                     | `^6.4`                     | 28.09.2023   | Feature Branch | master                                                                                          |\n| **2.x** | `10.0` - `10.6`            | `^5.4`                     | 19.12.2021   | No             | [2.x](https://github.com/dachcom-digital/pimcore-dynamic-search-data-provider-crawler/tree/2.x) |\n| **1.x** | `6.6` - `6.9`              | `^4.4`                     | 18.04.2021   | No             | [1.x](https://github.com/dachcom-digital/pimcore-dynamic-search-data-provider-crawler/tree/1.x) |\n\n***\n\n## Installation  \n```json\n\"require\" : {\n    \"dachcom-digital/dynamic-search\" : \"~3.0.0\",\n    \"dachcom-digital/dynamic-search-data-provider-crawler\" : \"~3.0.0\"\n}\n```\n\n### Dynamic Search Bundle\nYou need to install / enable the Dynamic Search Bundle first.\nRead more about it [here](https://github.com/dachcom-digital/pimcore-dynamic-search#installation).\nAfter that, proceed as followed:\n\nAdd Bundle to `bundles.php`:\n```php\n\u003c?php\n\nreturn [\n    \\DsWebCrawlerBundle\\DsWebCrawlerBundle::class =\u003e ['all' =\u003e true],\n];\n```\n\n***\n\n## Basic Setup\n\n```yaml\ndynamic_search:\n    context:\n        default:\n            data_provider:\n                service: 'web_crawler'\n                options:\n                    always:\n                        own_host_only: true\n                    full_dispatch:\n                        seed: 'http://your-domain.test'\n                        valid_links:\n                            - '@^http://your-domain.test.*@i'\n                        user_invalid_links:\n                            - '@^http://your-domain.test\\/members.*@i'\n                    single_dispatch:\n                        host: 'http://your-domain.test.test'\n                normalizer:\n                    service: 'web_crawler_localized_resource_normalizer'\n```\n\n***\n\n## Provider Options\n\n### always\n\n| Name                 | Default Value                    | Description |\n|:---------------------|:---------------------------------|:------------|\n| `own_host_only`      | false                            |             |\n| `allow_subdomains`   | false                            |             |\n| `allow_query_in_url` | false                            |             |\n| `allow_hash_in_url`  | false                            |             |\n| `allowed_mime_types` | ['text/html', 'application/pdf'] |             |\n| `allowed_schemes`    | ['http']                         |             |\n| `content_max_size`   | 0                                |             |\n\n### full_dispatch\n\n| Name                 | Default Value | Description |\n|:---------------------|:--------------|:------------|\n| `seed`               | null          |             |\n| `valid_links`        | []            |             |\n| `user_invalid_links` | []            |             |\n| `max_link_depth`     | 15            |             |\n| `max_crawl_limit`    | 0             |             |\n\n### single_dispatch\n\n| Name   | Default Value | Description |\n|:-------|:--------------|:------------|\n| `host` | null          |             |\n\n***\n\n## Resource Normalizer\n\n### DefaultResourceNormalizer\nIdentifier: `web_crawler_default_resource_normalizer`\nNormalize simple documents\nOptions: none\n\n### LocalizedResourceNormalizer\nIdentifier: `web_crawler_localized_resource_normalizer`\nScaffold localized documents\n\nOptions:\n\n| Name                           | Default Value                 | Allowed Type | Description                                                           |\n|:-------------------------------|:------------------------------|:-------------|:----------------------------------------------------------------------|\n| `locales`                      | all pimcore enabled languages | array        |                                                                       |\n| `skip_not_localized_documents` | true                          | bool         | if false, an exception rises if a document/object has no valid locale |\n\n***\n\n## Transformer\n\n### Scaffolder\n\n##### HttpResponseHtmlDataScaffolder\nIdentifier: `http_response_html_scaffolder`   \nSimple object scaffolder.   \nSupported types: `VDB\\Spider\\Resource` with content-type `text/html`.\n\n##### HttpResponsePdfDataScaffolder\nIdentifier: `http_response_pdf_scaffolder`   \nSimple object scaffolder.   \nSupported types: `VDB\\Spider\\Resource` with content-type `application/pdf`.\n\n##### PimcoreElementScaffolder\nIdentifier: `pimcore_element_scaffolder`   \nSimple object scaffolder.   \nSupported types: `Asset`, `Document`, `DataObject\\Concrete`.\n\n### Field Transformer\n\n##### UriExtractor\nIdentifier: `resource_uri_extractor`   \nSupported Scaffolder: `http_response_html_scaffolder`, `http_response_pdf_scaffolder`\n\nReturn Type: `string|null`   \nOptions: none\n\n##### LanguageExtractor\nIdentifier: `resource_language_extractor`   \nSupported Scaffolder: `http_response_html_scaffolder`, `http_response_pdf_scaffolder`\n\nReturn Type: `string|null`\nOptions: none\n\n##### MetaExtractor\nIdentifier: `resource_meta_extractor`   \nSupported Scaffolder: `http_response_html_scaffolder`\n\nReturn Type: `string|null`\nOptions: \n\n| Name   | Default Value | Allowed Type | Description                                      |\n|:-------|:--------------|:-------------|:-------------------------------------------------|\n| `name` | null          | string       | The name of the meta tag to fetch the value from |\n\n##### HtmlTagExtractor\nIdentifier: `resource_html_tag_content_extractor`   \nSupported Scaffolder: `http_response_html_scaffolder`\n\nReturn Type: `string|null`\nOptions: none\n\n##### TextExtractor\nIdentifier: `resource_text_extractor`   \nSupported Scaffolder: `http_response_html_scaffolder`, `http_response_pdf_scaffolder`\n\nReturn Type: `string|null`\n\n| Name                              | Default Value            | Allowed Type | Description                                              |\n|:----------------------------------|:-------------------------|:-------------|:---------------------------------------------------------|\n| `content_start_indicator`         | `\u003c!-- main-content --\u003e`  | string       | Marks the begin of the indexable page content            |\n| `content_end_indicator`           | `\u003c!-- /main-content --\u003e` | string       | Marks the end of the indexable page conten               |\n| `content_exclude_start_indicator` | null                     | null\\|string | Marks the begin of the text to be excluded from indexing |\n| `content_exclude_end_indicator`   | null                     | null\\|string | Marks the end of the text to be excluded from indexing   |\n\n##### TitleExtractor\nIdentifier: `resource_title_extractor`   \nSupported Scaffolder: `http_response_html_scaffolder`, `http_response_pdf_scaffolder`\n\nReturn Type: `string|null`\nOptions: none\n\n***\n\n## Copyright and License\nCopyright: [DACHCOM.DIGITAL](http://dachcom-digital.com)  \nFor licensing details please visit [LICENSE.md](./LICENSE.md)\n\n## Upgrade Info\nBefore updating, please [check our upgrade notes!](./UPGRADE.md)  \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdachcom-digital%2Fpimcore-dynamic-search-data-provider-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdachcom-digital%2Fpimcore-dynamic-search-data-provider-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdachcom-digital%2Fpimcore-dynamic-search-data-provider-crawler/lists"}