{"id":36998320,"url":"https://github.com/manofstrong/sitescrapper","last_synced_at":"2026-01-13T23:59:09.050Z","repository":{"id":62523426,"uuid":"214282507","full_name":"manofstrong/sitescrapper","owner":"manofstrong","description":"A PHP library to Scrape Websites from their Sitemaps and Extract Relevant Content from the Webpage and Upload to a Database","archived":false,"fork":false,"pushed_at":"2019-10-13T13:20:19.000Z","size":115,"stargazers_count":6,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-23T18:29:34.514Z","etag":null,"topics":["keywords-extraction","php","scraper","scraping-websites","sitemap-xml","text-extraction"],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/manofstrong.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-10-10T20:46:27.000Z","updated_at":"2022-02-21T14:27:17.000Z","dependencies_parsed_at":"2022-11-02T14:15:22.791Z","dependency_job_id":null,"html_url":"https://github.com/manofstrong/sitescrapper","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/manofstrong/sitescrapper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manofstrong%2Fsitescrapper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manofstrong%2Fsitescrapper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manofstrong%2Fsitescrapper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manofstrong%2Fsitescrapper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/manofstrong","download_url":"https://codeload.github.com/manofstrong/sitescrapper/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manofstrong%2Fsitescrapper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28406238,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-13T21:51:37.118Z","status":"ssl_error","status_checked_at":"2026-01-13T21:45:14.585Z","response_time":56,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["keywords-extraction","php","scraper","scraping-websites","sitemap-xml","text-extraction"],"created_at":"2026-01-13T23:59:08.428Z","updated_at":"2026-01-13T23:59:09.045Z","avatar_url":"https://github.com/manofstrong.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Site Scrapper\nA PHP library to scrape websites based on their sitemaps and extract relevant content from the webpages the content is then uploaded a database for later use.\n\nThe [Sitemaps.org](http://www.sitemaps.org/) protocol is the leading standard and is supported by Google, Bing, Yahoo, Ask and many others and has now become an accepted means of displaying webpages in a website and their relevance. This has meant that most of the modern websites have implemented sitemaps which makes it easier for a web scrapper to avoid unnecessary measures to find the links and go straight to the source. Please note that the library can recursively parse through sitemaps so only one sitemap per website is needed. \n\nThis library also eliminates the need for exploring site specific html tags in search of the relevant content by stripping down the page content to the most important parts of the page by removing boilerplate and extracting the rest as full text. The library then obtains the keywords of the extracted text and the word count of the text. These are essential for later data analysis and keywords. Finally, the library uploads the content into MySQL database whose schema has been provided in the database folder of this project. \n\nBasically, this is a blind bulk scrapping tool, just provide it with a list of sitemaps, run it. It will run for as long as it takes to scrape through the pages of the websites provided and upload them to the database for you. \n\n**This library is designed to be run from the Command Line rather than web browser. Please consider it a CLI tool and use it as such.**\n\n## Features\n- Sitemap parsing (either a single site, or a list of sites)\n- Scrapping (relevant content extraction)\n- Keyword extraction\n- Word count of extracted data\n- Custom User-Agent string\n- Database uploading of extracted content\n\n## Sitemap Formats supported\n- XML `.xml`\n- Compressed XML `.xml.gz`\n- Robots.txt rule sheet `robots.txt`\n\n## Webpage Formats supported\n- HTML `text/html`\n\n## Sitemap File Formats supported\n- Text `text/txt`\n\n## Installation\nThe library is available for install via [Composer](https://getcomposer.org). Just add this to your `composer.json` file:\n```json\n{\n    \"require\": {\n        \"manofstrong/sitescrapper\": \"^0.0.1\"\n    }\n}\n```\nThen run `composer update`.\n\n## Getting Started\n\n### Basic display example\nReturns the content of a specified number of pages from a single sitemap. Does not store into database.\n\n```php\n\u003c?php\nrequire 'vendor/autoload.php';\n\nuse Manofstrong\\sitescrapper;\n$scrapeThisSite = new SiteScrapper();\n\n$singleSitemapUrl = 'https://www.php.com/sitemap.xml';   // can be .xml or .xml.gz or robots.txt file\n$numberOfPages = 2;\t// must be a digit without single or double quotation marks: '2' or \"2\" will fail.\n \n$scrapeThisSite -\u003e showContentSiteMap($singleSitemapUrl, $numberOfPages);\n\n```\nThis command displays the content of each page as an array with the four elements the url, the title, the keywords, the content, and the word count as shown in the image below:\n\n![An image of the method showContentSiteMap() displaying of the two pages as arrays](https://raw.githubusercontent.com/manofstrong/sitescrapper/master/public/basicexample.png)\n### A single sitemap\nMethod to extract the webpages from a single sitemap. This will go through the sites and update the database provided. It will return the url and completion status. \n```php\n\u003c?php\nrequire 'vendor/autoload.php';\n\nuse Manofstrong\\sitescrapper;\n$scrapeThisSite = new SiteScrapper();\n\n$singleSitemapUrl = 'https://www.php.com/sitemap.xml';\n \n// MySQL table structure in the database folder of this repository\n$scrapeThisSite -\u003e databaseCredentials('database','host','username','password'); \n\n$scrapeThisSite -\u003e singleSiteMap($singleSitemapUrl);\n```\n### Array of sitemaps\nMethod to extract the webpages from sitemaps provided via array. This will go through the sites in the array, and update the database provided. It will return the url and completion status. This is advisable if you have less than 10 different sites that you want to scrape.\n```php\n\u003c?php\nrequire 'vendor/autoload.php';\n\nuse Manofstrong\\sitescrapper;\n$scrapeThisSite = new SiteScrapper();\n\n$sitemapArray = ['https://site1.com/sitemap.xml','https://site2.com/sitemap.xml','https://site3.com/sitemap.xml'];\n\n// MySQL table structure in the database folder of this repository\n$scrapeThisSite -\u003e databaseCredentials('database','host','username','password'); \n\n$scrapeThisSite -\u003e siteMapsArray($sitemapArray);\n```\n### File of Sitemaps\nMethod to extract the webpages from sitemaps provided via array. This will go through the all sites in the file, and update the database provided. It will return the url and completion status. This is advisable if you have more than 10 different sites that you want to scrape.\n\nThe file must must a text file and the each sitemap must be in its own line without any other text. The urls must also be in compliance with the `[RFC 2396]`(https://www.ietf.org/rfc/rfc2396.txt).\n\nFor the sitemap urls that are not compliant, the library will create a file called `skippedeurls.txt` and list them there. You can then go through this file making  the corrections needed to ensure compliance. \n```php\n\u003c?php\nrequire 'vendor/autoload.php';\n\nuse Manofstrong\\sitescrapper;\n$scrapeThisSite = new SiteScrapper();\n\n$sitemapFile  = __DIR__ . '/yourtextfile.txt';\n \n// MySQL table structure in the database folder of this repository\n$scrapeThisSite -\u003e databaseCredentials('database','host','username','password'); \n\n$scrapeThisSite -\u003e siteMapFile($sitemapFile);\n```\n\n**Sample file structure**\n\n    https://site1.com/sitemap.xml\n    https://site2.com/sitemap.xml\n    http://site3.com/sitemap.xml\n    https://site4.com/sitemap.xml\n    https://site5.com/sitemap.xml\n    https://site6.com/sitemap.xml\n    https://site7.com/sitemap.xml\n    https://site8.com/sitemap.xml\n    https://site9.com/sitemap.xml\n    https://site10.com/sitemap.xml\n    https://www.site11.com/sitemap.xml\n    https://www.site12.com/sitemap.xml\n    http://www.site13.com/sitemap.xml\n    https://www.site14.com/sitemap.xml\n    https://www.site15.com/sitemap.xml\n    https://www.site16.com/sitemap.xml\n    https://www.site17.com/sitemap.xml\n    https://www.site18.com/sitemap.xml\n    https://www.site19.com/sitemap.xml\n    https://www.site20.com/sitemap.xml\n    http://site21.com/sitemap.xml\n\n### Advanced setting of setting User Agent\n__Note:__ The default User Agent is `ManofStrong SiteScrapper Tool v1.0 (+https://github.com/manofstrong/sitescrapper/)`\n\nIf you want to change and have your own just use this method.\n```php\n\u003c?php\nrequire 'vendor/autoload.php';\n\nuse Manofstrong\\sitescrapper;\n$scrapeThisSite = new SiteScrapper();\n\n$scrapeThisSite -\u003e setUserAgent('Example User Agent. For Text Data Data Assigment.');\n\n```\n\n## License:\nSiteScrapper is under the MIT license.\n\n## Limitations:\nThis is not a precise data collection tools especially for those who want to obtain specific sections of a webpage to the exclusion of others. This library simply collects the most important section of the webpage such as large blocks of text. Therefore, this library best works for someone with the goal of massive text data collection and extraction for big data or other analytical research.\n\n## Usage Tip:\nIf you are running this on a remote server that you have ssh into, run it in a Screen to avoid interruptions. Learn more about the [Linux Screen Command here](https://linuxize.com/post/how-to-use-linux-screen/).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmanofstrong%2Fsitescrapper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmanofstrong%2Fsitescrapper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmanofstrong%2Fsitescrapper/lists"}