{"id":18810463,"url":"https://github.com/crispy-computing-machine/supersimplecrawler","last_synced_at":"2025-04-13T20:31:02.072Z","repository":{"id":171910778,"uuid":"648586769","full_name":"crispy-computing-machine/SuperSimpleCrawler","owner":"crispy-computing-machine","description":"Super Simple Crawler for PHP 8.3.2","archived":false,"fork":false,"pushed_at":"2024-09-28T07:47:42.000Z","size":54,"stargazers_count":1,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-10-28T16:10:33.913Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/crispy-computing-machine.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-02T10:14:09.000Z","updated_at":"2024-09-28T07:47:23.000Z","dependencies_parsed_at":null,"dependency_job_id":"b2257bd6-3429-4b4d-a320-3f1cc4708ceb","html_url":"https://github.com/crispy-computing-machine/SuperSimpleCrawler","commit_stats":null,"previous_names":["crispy-computing-machine/supersimplecrawler"],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crispy-computing-machine%2FSuperSimpleCrawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crispy-computing-machine%2FSuperSimpleCrawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crispy-computing-machine%2FSuperSimpleCrawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crispy-computing-machine%2FSuperSimpleCrawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/crispy-computing-machine","download_url":"https://codeload.github.com/crispy-computing-machine/SuperSimpleCrawler/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223603336,"owners_count":17172086,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T23:20:30.487Z","updated_at":"2024-11-07T23:20:30.844Z","avatar_url":"https://github.com/crispy-computing-machine.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SuperSimpleCrawler\nSuper Simple Crawler for PHP 8\n[![Latest Stable Version](https://poser.pugx.org/crispy-computing-machine/SuperSimpleCrawler/v/stable)](https://packagist.org/packages/crispy-computing-machine/SuperSimpleCrawler) [![Total Downloads](https://poser.pugx.org/bcrispy-computing-machine/SuperSimpleCrawler/downloads)](https://packagist.org/packages/crispy-computing-machine/SuperSimpleCrawler)\n\n```\ncomposer require crispy-computing-machine/supersimplecrawler\n```\n\n```php\n//Composer\nrequire 'vendor/autoload.php';\n\n// Libs\nuse SuperSimple\\Crawler;\nuse SuperSimple\\CrawlerCompleteException;\nuse DOMDocument;\n\n// Configure the crawler.\n$crawler = new Crawler($verbose = true);\ntry {\n    $crawler-\u003esetUrl(\"https://www.php.net/\"); // Set the URL.\n    $crawler-\u003esetPort(80); // Set the port (80 is the default HTTP port).\n    $crawler-\u003esetFollowRedirects(true); // Follow redirects.\n    $crawler-\u003esetFollowMode(2); // Follow only links within the same host.\n    $crawler-\u003esetRequestLimit(10); // Limit the number of requests.\n    $crawler-\u003esetContentSizeLimit(2000000); // Limit the content size (2 MB in this case).\n    $crawler-\u003esetTrafficLimit(10000000); // Limit the traffic (10 MB in this case).\n    $crawler-\u003esetUserAgentString(\"Mozilla/5.0 (compatible; MyCrawler/1.0)\"); // Set a custom user agent.\n    $crawler-\u003esetWorkingDirectory(sys_get_temp_dir() . DIRECTORY_SEPARATOR . 'crawler'); // Set a directory for storing data.\n    $crawler-\u003esetProxy(); // Set a proxy (if required).\n    $crawler-\u003esetCertificateVerify(true); // Verify SSL certificates.\n    $crawler-\u003esetConnectionTimeout(10); // Set the connection timeout (10 seconds).\n    $crawler-\u003esetStreamTimeout(20); // Set the stream timeout (20 seconds).\n    $crawler-\u003esetRequestDelay(3000); // Wait 1048576 millisecond between requests.\n    $crawler-\u003esetConcurrency(10); // multiprocessing mode\n    $crawler-\u003eaddContentTypeReceiveRule(['text/html', 'application/json']);\n    \n    // Handle doc\n    $crawler-\u003esetFulfilledCallback(function($url, DOMDocument $content){\n        echo 'Successful request! ' . $url . ' -\u003e ' . strlen($content-\u003etextContent) . PHP_EOL;\n    });\n\n    // Handle error\n    $crawler-\u003esetRejectedCallback(function ($url, DOMDocument $content){\n        echo 'Failed request! ' . $url . ' -\u003e ' . strlen($content-\u003etextContent) . PHP_EOL;\n    });\n\n    // Start the crawling process.\n    $crawler-\u003ecrawl();\n\n} catch (CrawlerCompleteException $e) {\n\n    // Crawler complete report\n    Crawler::log(\"Total pages downloaded: \" . $e-\u003egetTotalPages(), \"info\");\n    Crawler::log(\"Total size downloaded: \" . $e-\u003egetTotalSize() . \" bytes\", \"info\");\n    Crawler::log(\"Total links followed: \" . $e-\u003egetTotalLinks(), \"info\");\n    Crawler::log(\"Abort Reason: \" . $e-\u003egetMessage(), \"info\");\n\n} finally {\n    // Additional clean up or summary actions\n    Crawler::log(\"Crawler has ended gracefully.\", \"success\");\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcrispy-computing-machine%2Fsupersimplecrawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcrispy-computing-machine%2Fsupersimplecrawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcrispy-computing-machine%2Fsupersimplecrawler/lists"}