{"id":13464675,"url":"https://github.com/bplawler/crawler","last_synced_at":"2025-03-25T11:32:02.089Z","repository":{"id":1881986,"uuid":"2807639","full_name":"bplawler/crawler","owner":"bplawler","description":"Scala DSL for web crawling","archived":false,"fork":false,"pushed_at":"2016-08-02T21:33:54.000Z","size":84,"stargazers_count":148,"open_issues_count":2,"forks_count":40,"subscribers_count":14,"default_branch":"master","last_synced_at":"2024-10-29T17:52:28.410Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bplawler.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2011-11-19T06:31:24.000Z","updated_at":"2024-09-21T12:53:36.000Z","dependencies_parsed_at":"2022-07-31T11:48:05.833Z","dependency_job_id":null,"html_url":"https://github.com/bplawler/crawler","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bplawler%2Fcrawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bplawler%2Fcrawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bplawler%2Fcrawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bplawler%2Fcrawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bplawler","download_url":"https://codeload.github.com/bplawler/crawler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245454110,"owners_count":20617978,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T14:00:48.477Z","updated_at":"2025-03-25T11:31:57.067Z","avatar_url":"https://github.com/bplawler.png","language":"Scala","funding_links":[],"categories":["All","Scala"],"sub_categories":[],"readme":"# crawler - A DSL for web crawling in Scala\n\nThe purpose of this project is to provide a nice DSL wrapper around the\ncumbersome htmlunit Java library.  Here is an example taken from a unit \ntest in this package:\n\n    import crawler._\n\n    class TestCrawler(output: java.io.OutputStream) extends Crawler {\n      var result = \"\"\n      def crawl = {\n        navigateTo(\"http://www.google.com\") {\n          in(form having id(\"tsf\")) {\n            in(textField having id(\"lst-ib\")) {\n              typeIn(\"bplawler\")\n            }\n            in(submit having name(\"btnK\")) {\n              click ==\u003e\n            }\n          }\n        }\n        onCurrentPage {\n          result = from(div having id(\"resultStats\")) getTextContent\n          val url = from(\n            anchor having xPath(\"//div[@id='field_timetable_file-wrapper']/a\")\n          ).getAttributes.getNamedItem(\"href\").getTextContent\n          download(url).writeTo(output)\n        }\n      }\n    }\n\nThis `TestCrawler` class defines a crawl that will navigate to google, \nfind the form whose id is `tsf`, type something into the form, then\nclick on the submit button named `btnK`, which will then take us to a \nnew page (the search results) where we can then grab the content of the\n`resultStats` div.\n\nIt also grabs URL from a link defined by XPath and downloads it to given\nOutputStream.\n\nAlternatively you could just get bytes instead of writing downloaded data to \nOutputStream: `download(url).getBytes`.\n\n## Background\n\nThis DSL was created to simplify the code needed to programmatically access\nweb pages and do something meaningful with the content.  It is backed by the\nJava [HtmlUnit](http://htmlunit.sourceforge.net/) library, which, according to\ntheir web site, provides a \"GUI-less browser for Java programs.\"  The library\nis very good at what it does, but I found that using it generally resulted in\ncode that was pretty difficult to read.\n\nThis DSL is also my first attempt to write such a thing in Scala, so really \nthis is just sort of an academic project to learn as much as I can about Scala\nand about writing DSL's in Scala.  There are a few brittle areas in this thing\nthat could greatly benefit from some clear error handling, but for what I \nwas trying to do, the code here did the trick just fine.\n\n## Basic Language Structure\n\nThe first part of any web crawl is to provide a starting point.  This is \ndone with the `navigateTo` method.  `navigateTo` takes a string and is \nfollowed by the code block that contains the stuff you want to do with\nthis page.\n\n    navigateTo(\"http://www.google.com\") { ... }\n\nInside the code block, you can use the DSL keywords to find individual HTML\nelements and do operations on those things.  On of the more common keywords is\n`in`, which receives as an argument a bit of code that identifies an HTML\nnode on the current page, then opens another code block to do processing within\nthat HTML node.  The following excerpt will navigate to the google home page\nand find the form element that has an id of `tsf`.\n\n    navigateTo(\"http://www.google.com\") {\n      in(form having id(\"tsf\")) { ... }\n    }\n\nThe code block after the `in` call will operate on the form element that was\nfound.  If there was no form with that id on the page, you'll get an error \nbut it will be the one that is generated by HtmlUnit - I have not yet made \nany effort to wrap the errors nicely.  Inside the code block for this form,\nwe can do things like access the individual input fields and enter in \nvalues.\n\n    navigateTo(\"http://www.google.com\") {\n      in(form having id(\"tsf\")) {\n        in(textField having id(\"lst-ib\")) {\n          typeIn(\"bplawler\")\n        }\n        ...\n      }\n    }\n\nHere we have expanded the example to find the text field on the Google home\npage and type in a search term.  With this typed in, the next thing we will\nwant to do is submit the form and do our search.\n\n    navigateTo(\"http://www.google.com\") {\n      in(form having id(\"tsf\")) {\n        in(textField having id(\"lst-ib\")) {\n          typeIn(\"bplawler\")\n        }\n        in(submit having name(\"btnK\") {\n          click ==\u003e\n        }\n      }\n    }\n\nClicking the button is as easy as finding the submit button in the HTML and\ncalling `click`.  But what is that wierd `==\u003e` operator?  It turns out that\nthis click on our GUI-less browser will take us to a new web page.  The \n`==\u003e` operator without an argument signifies that this new page is the next\npage we will be working with.  So rather than having to use `navigateTo` \nagain, we can simply end this code block and use the `onCurrentPage` method\nto start our next code block.\n\n    navigateTo(\"http://www.google.com\") { ... }\n    onCurrentPage {\n      result = from(div having id(\"resultStats\")) getTextContent\n    }\n\nIn this example, what we are doing is using the `from` keyword to find a \nparticular HTML element (just as with `in`) but this time we are going to \nget something out of the element and put that value in a variable.  Remember \nthat this DSL is just an extension of Scala, and that we could also just as\neasily now call out to another method from within here and do some meaningful\nwork.  One other keyword that is supported is `forAll` which receives an\n[XPath](http://www.w3schools.com/xpath/) and a subsequent code block over\nwhich all of the items in the list will be run.\n\n    navigateTo(\"http://www.google.com\") { ... }\n    onCurrentPage {\n      result = from(div having id(\"resultStats\")) getTextContent\n      \n      forAll(div having xPath(\"\"\"//ol[@id = \"rso\"]/li/div[@class = \"vsc\"]\"\"\")) {\n        println(from(anchor having xPath(\"h3/a\")) getTextContent)\n      }\n    }\n\nThis invocation of `forAll` will loop through each individual search result\nand print out the main anchor text for each.\n\n## Releases\n\n* 0.6.0 (2013.08.16)\n  * Switched to Scala 2.10.2 for building (with 2.10 binary compatibility).\n  * Added #download method.\n* 0.5.0 (2012.06.24)\n  * Bumping the version number to something that is completely unrelated \n    to circupon.\n  * During crawler construction, it is now possible to set whether \n    css is supported (default is false) and whether JavaScript is supported\n    (default is true).\n* 0.3.3 (2012.06.15)\n  * Added support for `mouseOver` in the DSL.  Works just like `click`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbplawler%2Fcrawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbplawler%2Fcrawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbplawler%2Fcrawler/lists"}