{"id":18649780,"url":"https://github.com/ashwanthkumar/scraphp","last_synced_at":"2025-11-05T10:30:30.656Z","repository":{"id":2424117,"uuid":"3392899","full_name":"ashwanthkumar/scraphp","owner":"ashwanthkumar","description":"Web Crawler daemon in PHP, use XPath to get content into objects and persist them. ","archived":false,"fork":false,"pushed_at":"2017-04-19T05:15:26.000Z","size":309,"stargazers_count":12,"open_issues_count":1,"forks_count":4,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-12-27T12:44:49.740Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://blog.ashwanthkumar.in/2012/02/introducing-scraphp-web-crawler-in-php.html","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ashwanthkumar.png","metadata":{"files":{"readme":"README.markdown","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-02-09T00:27:25.000Z","updated_at":"2019-12-18T10:36:42.000Z","dependencies_parsed_at":"2022-09-03T21:21:15.510Z","dependency_job_id":null,"html_url":"https://github.com/ashwanthkumar/scraphp","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashwanthkumar%2Fscraphp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashwanthkumar%2Fscraphp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashwanthkumar%2Fscraphp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashwanthkumar%2Fscraphp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ashwanthkumar","download_url":"https://codeload.github.com/ashwanthkumar/scraphp/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239456405,"owners_count":19641843,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T06:40:34.012Z","updated_at":"2025-11-05T10:30:30.611Z","avatar_url":"https://github.com/ashwanthkumar.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![No Maintenance Intended](http://unmaintained.tech/badge.svg)](http://unmaintained.tech/)\n\nIntroduction\n------------\nScraphp (say Scraph, last p is silent) is a web crawling program. It is basically built to be a standalone executable which can crawl websites and store extract useful content out of it. I created this script for a challenge posted by Indix on Jan 2012, where in I was asked to crawl AGMarket (http://agmarknet.nic.in/) to get the prices of all the products, and store their prices. I also had to version the prices such that it should persist across dates. \n\nScraph was inspired from a similar project called Scrappy, written in Python. This is not an attempt to port it, but just wanted to see how much similar properties can I build from it in less than a day. \n\nOne of the major features I would like to call it is, When you crawl the page you can extract entites out of it based on XPath. So basically when we crawl a page I create a bean whose properties are set of values got by applying the given XPath on the page. Each XPath is completely independent of the other. Currently Scraph supports creating only 1 type of object per page. \n\nHack into the source code, its well commented and easy to modify as per requirement. All the details of the crawling page, XPath queries are all provided in the configuration.php or you can supply your own config file, see the Usage below. \n\nDirectory Layout\n-----------------\n\n`./data/`\t\t-\tContains the log and default scrapper.db SQLite databases\n\n`./lib/`\t\t-\tContains the librarys that were used in the application\n\n`./logs/`\t\t-\tContains the logs that are generated by the application. \n\n`./model/`\t\t-\tContains the models (Beans) and interfaces used in the application\n\n\nUsage\n-----\n\nDefault configuration implemenation crawls the AGMarket (initial question) and adds the content to the datastore\n\n`$ ./scraphp`\n\nI have created a sample config for crawling Flipkart also which can be executed as\n\n`$ ./scraphp --configuration=config.flipkart.sample.php`\n\nThis crawls all the products with their name, type (major classifications like books, computers, etc.) and price. \n\nConfiguration Options\n--------------------\n\nConfiguration layout is well documented in the samples provided.\n\nGeneral Method to use\n---------------------\n\n1.\tCopy any of the sample existing configuration and update the values of _bean and XPath\n2.\tAlso create a class which implements the Scrapable interface and implement the method save();\n3.\tAll the `props` that were defined in the configuration are available as properties in the object. \u003cbr /\u003e\n\tIf the prop item is not found, it will not be available, always use isset($this-\u003eitem) to check \n\tif the item was crawled from the page.\n4.\tFor datastore we are using RedBeanPHP (http://www.redbeanphp.com/manual/), for its simplicity in \n\tdesign. A 15 min read of the documentation can get you started in RedBean right away :-)\n5.\tOnce the configuration and the corresponding beans are created, you can just invoke the spider as \n\tthe normal\texecutable. \n\t\nSpecial Mention\n---------------\n\n*\tWhen we crawl any page we use Tidy to clean the document and then load its DOM. So normal Google Chrome's or Firebug's XPath might not work on the way. Use YQL to test the XPath you are using (YQL also tidies the document). Sample YQL query might be - http://goo.gl/3NDlm\n\nDepencies\n---------\n\nScrapper depends on \n\n* php5-tidy, \n* php5-mysql, \n* php5-sqlite, \n* php5-cli \n\nWebserver configured with PHP5 as CGI or natively as a module (like Apache) to use the Web interface for querying the loaded data. \nPHP5 packages.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fashwanthkumar%2Fscraphp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fashwanthkumar%2Fscraphp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fashwanthkumar%2Fscraphp/lists"}