{"id":13463812,"url":"https://github.com/zrashwani/arachnid","last_synced_at":"2026-01-13T22:48:17.430Z","repository":{"id":13001140,"uuid":"15680337","full_name":"zrashwani/arachnid","owner":"zrashwani","description":"Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites","archived":false,"fork":false,"pushed_at":"2022-09-10T20:27:13.000Z","size":221,"stargazers_count":254,"open_issues_count":5,"forks_count":59,"subscribers_count":20,"default_branch":"master","last_synced_at":"2025-11-13T18:26:24.659Z","etag":null,"topics":["crawler","php","scraping","seo"],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zrashwani.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-01-06T17:32:41.000Z","updated_at":"2025-10-18T01:06:29.000Z","dependencies_parsed_at":"2022-08-07T07:00:56.474Z","dependency_job_id":null,"html_url":"https://github.com/zrashwani/arachnid","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"purl":"pkg:github/zrashwani/arachnid","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zrashwani%2Farachnid","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zrashwani%2Farachnid/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zrashwani%2Farachnid/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zrashwani%2Farachnid/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zrashwani","download_url":"https://codeload.github.com/zrashwani/arachnid/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zrashwani%2Farachnid/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28403726,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-13T14:36:09.778Z","status":"ssl_error","status_checked_at":"2026-01-13T14:35:19.697Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","php","scraping","seo"],"created_at":"2024-07-31T14:00:28.872Z","updated_at":"2026-01-13T22:48:17.405Z","avatar_url":"https://github.com/zrashwani.png","language":"PHP","funding_links":[],"categories":["Crawlers","PHP"],"sub_categories":["Gists"],"readme":"# Arachnid Web Crawler\r\n\r\nThis library will crawl all unique internal links found on a given website\r\nup to a specified maximum page depth.\r\n\r\nThis library is using [_symfony/panther_](https://github.com/symfony/panther) \u0026 [FriendsOfPHP/Goutte](https://github.com/FriendsOfPHP/Goutte) libraries to scrap site pages and extract main SEO-related info, including: \r\n`title`, `h1 elements`, `h2 elements`, `statusCode`, `contentType`, `meta description`, `meta keyword` and `canonicalLink`.\r\n\r\n\r\nThis library is based on the original blog post by Zeid Rashwani here:\r\n\r\n\u003chttp://zrashwani.com/simple-web-spider-php-goutte\u003e\r\n\r\nJosh Lockhart adapted the original blog post's code (with permission)\r\nfor Composer and Packagist and updated the syntax to conform with\r\nthe PSR-2 coding standard.\r\n\r\n[![Build Status](https://travis-ci.com/zrashwani/arachnid.svg?branch=master)](https://travis-ci.com/zrashwani/arachnid)\r\n[![codecov](https://codecov.io/gh/zrashwani/arachnid/branch/master/graph/badge.svg)](https://codecov.io/gh/zrashwani/arachnid)\r\n\r\n## Sponsored By\r\n[\u003cimg src=\"./oxylabs.png\" width=\"300px\"/\u003e](https://oxylabs.go2cloud.org/aff_c?offer_id=7\u0026aff_id=447\u0026url_id=32)\r\n\r\n## How to Install\r\n\r\nYou can install this library with [Composer][composer]. Drop this into your `composer.json`\r\nmanifest file:\r\n\r\n    {\r\n        \"require\": {\r\n            \"zrashwani/arachnid\": \"dev-master\"\r\n        }\r\n    }\r\n\r\nThen run `composer install`.\r\n\r\n## Getting Started\r\n\r\n### Basic Usage:\r\nHere's a quick demo to crawl a website:\r\n```php\r\n    \u003c?php\r\n    require 'vendor/autoload.php';\r\n\r\n    $url = 'http://www.example.com';\r\n    $linkDepth = 3;\r\n    // Initiate crawl, by default it will use http client (GoutteClient), \r\n    $crawler = new \\Arachnid\\Crawler($url, $linkDepth);\r\n    $crawler-\u003etraverse();\r\n\r\n    // Get link data\r\n    $links = $crawler-\u003egetLinksArray(); //to get links as objects use getLinks() method\r\n    print_r($links);\r\n```\r\n\r\n### Enabling Headless Browser mode:\r\n\r\nHeadless browser mode can be enabled, so it will use Chrome engine in the background which is useful to get contents of JavaScript-based sites.\r\n\r\n`enableHeadlessBrowserMode` method set the scraping adapter used to be `PantherChromeAdapter` which is based on [Symfony Panther](https://github.com/symfony/panther) library: \r\n```php\r\n    $crawler = new \\Arachnid\\Crawler($url, $linkDepth);\r\n    $crawler-\u003eenableHeadlessBrowserMode()\r\n            -\u003etraverse()\r\n            -\u003egetLinksArray();\r\n```\r\n\r\nIn order to use this, you need to have [chrome-driver](https://sites.google.com/a/chromium.org/chromedriver/) installed on your machine, you can use `dbrekelmans/browser-driver-installer` to install chromedriver locally: \r\n```\r\ncomposer require --dev dbrekelmans/bdi\r\n./vendor/bin/bdi driver:chromedriver drivers\r\n```\r\n    \r\n## Advanced Usage:\r\n\r\n   Set additional options to underlying http client, by specifying array of options in constructor \r\nor creating Http client scrapper with desired options:\r\n\r\n```php\r\n    \u003c?php\r\n        use \\Arachnid\\Adapters\\CrawlingFactory;\r\n        //third parameter is the options used to configure http client\r\n        $clientOptions = ['auth_basic' =\u003e array('username', 'password')];\r\n        $crawler = new \\Arachnid\\Crawler('http://github.com', 2, $clientOptions);\r\n           \r\n        //or by creating and setting scrap client\r\n        $options = array(\r\n            'verify_host' =\u003e false,\r\n            'verify_peer' =\u003e false,\r\n            'timeout' =\u003e 30,\r\n        );\r\n                        \r\n        $scrapperClient = CrawlingFactory::create(CrawlingFactory::TYPE_HTTP_CLIENT, $options);\r\n        $crawler-\u003esetScrapClient($scrapperClient);\r\n```\r\n\r\n   You can inject a [PSR-3][psr3] compliant logger object to monitor crawler activity (like [Monolog][monolog]):\r\n```php\r\n    \u003c?php    \r\n    $crawler = new \\Arachnid\\Crawler($url, $linkDepth); // ... initialize crawler   \r\n\r\n    //set logger for crawler activity (compatible with PSR-3)\r\n    $logger = new \\Monolog\\Logger('crawler logger');\r\n    $logger-\u003epushHandler(new \\Monolog\\Handler\\StreamHandler(sys_get_temp_dir().'/crawler.log'));\r\n    $crawler-\u003esetLogger($logger);\r\n    ?\u003e\r\n```\r\n\r\n   You can set crawler to visit only pages with specific criteria by specifying callback closure using `filterLinks` method:\r\n\r\n```php\r\n    \u003c?php\r\n    //filter links according to specific callback as closure\r\n    $links = $crawler-\u003efilterLinks(function($link) {\r\n                        //crawling only links with /blog/ prefix\r\n                        return (bool)preg_match('/.*\\/blog.*$/u', $link); \r\n                    })\r\n                    -\u003etraverse()\r\n                    -\u003egetLinks();\r\n\r\n```    \r\n    \r\n   You can use `LinksCollection` class to get simple statistics about the links, as following:\r\n```php\r\n    \u003c?php\r\n    $links = $crawler-\u003etraverse()\r\n                     -\u003egetLinks();\r\n    $collection = new LinksCollection($links);\r\n\r\n    //getting broken links\r\n    $brokenLinks = $collection-\u003egetBrokenLinks();\r\n   \r\n    //getting links for specific depth\r\n    $depth2Links = $collection-\u003egetByDepth(2);\r\n\r\n    //getting external links inside site\r\n    $externalLinks = $collection-\u003egetExternalLinks();\r\n```\r\n\r\n## How to Contribute\r\n\r\n1. Fork this repository\r\n2. Create a new branch for each feature or improvement\r\n3. Apply your code changes along with corresponding unit test\r\n4. Send a pull request from each feature branch\r\n\r\nIt is very important to separate new features or improvements into separate feature branches,\r\nand to send a pull request for each branch. This allows me to review and pull in new features\r\nor improvements individually.\r\n\r\nAll pull requests must adhere to the [PSR-2 standard][psr2].\r\n\r\n## System Requirements\r\n\r\n* PHP 7.2.0+\r\n\r\n## Authors\r\n\r\n* Josh Lockhart \u003chttps://github.com/codeguy\u003e\r\n* Zeid Rashwani \u003chttp://zrashwani.com\u003e\r\n\r\n## License\r\n\r\nMIT Public License\r\n\r\n[composer]: http://getcomposer.org/\r\n[psr2]: https://github.com/php-fig/fig-standards/blob/master/accepted/PSR-2-coding-style-guide.md\r\n[psr3]: https://github.com/php-fig/fig-standards/blob/master/accepted/PSR-3-logger-interface.md\r\n[monolog]: https://github.com/Seldaek/monolog\r\n\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzrashwani%2Farachnid","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzrashwani%2Farachnid","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzrashwani%2Farachnid/lists"}