{"id":19973810,"url":"https://github.com/7bridges-eu/shelob","last_synced_at":"2025-05-04T02:32:15.176Z","repository":{"id":47553858,"uuid":"196012382","full_name":"7bridges-eu/shelob","owner":"7bridges-eu","description":"Clojure scraping framework wanna-be","archived":false,"fork":false,"pushed_at":"2022-10-08T10:15:09.000Z","size":149,"stargazers_count":5,"open_issues_count":1,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-08T01:42:17.761Z","etag":null,"topics":["clojure","framework","jsoup","scraping","selenium"],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/7bridges-eu.png","metadata":{"files":{"readme":"README.adoc","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-07-09T13:16:54.000Z","updated_at":"2024-03-01T20:39:18.000Z","dependencies_parsed_at":"2022-09-23T13:20:37.779Z","dependency_job_id":null,"html_url":"https://github.com/7bridges-eu/shelob","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/7bridges-eu%2Fshelob","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/7bridges-eu%2Fshelob/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/7bridges-eu%2Fshelob/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/7bridges-eu%2Fshelob/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/7bridges-eu","download_url":"https://codeload.github.com/7bridges-eu/shelob/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252279025,"owners_count":21722816,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clojure","framework","jsoup","scraping","selenium"],"created_at":"2024-11-13T03:13:00.185Z","updated_at":"2025-05-04T02:32:14.949Z","avatar_url":"https://github.com/7bridges-eu.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"Shelob\n------\n\nShelob wraps https://www.seleniumhq.org/[Selenium] to let you browse a website\nand scrape its contents.\n\nRationale\n~~~~~~~~~\n\nSelenium automates web browsing, primarily for testing and administration\npurposes.\n\nShelob wraps Selenium to make it more idiomatic and adherent to the Clojure way\nof coding, and exposes facilities to scrape web pages.\n\nVersion\n~~~~~~~\n\nimage:https://img.shields.io/clojars/v/eu.7bridges/shelob.svg[link=\"https://clojars.org/eu.7bridges/shelob\"]\n\nExample\n~~~~~~~\n\nA simple DuckDuckGo search:\n\n* Type \"clojure\" on the search field\n* Click on the magnify glass to perform the search\n* Retrieve the URLs of the visible results\n\n[source,clojure]\n----\n(require '[shelob.core :as sh])\n(require '[shelob.browser :as shb])\n(require '[shelob.scraper :as shs])\n\n(def context\n  {:driver-options {:browser :firefox}\n   :pool-size 2\n   :init-messages [{:msg :go :url \"https://duckduckgo.com/html/\"}]})\n\n(defn scrape-result\n  [document]\n  (println (map \n            #(shs/attribute % \"href\") \n            (shs/select document \".result__a\"))))\n\n(defn example\n  []\n  (sh/init context)\n  (let [msg [{:msg :fill\n              :locator (shb/by-css-selector \"#search_form_input_homepage\")\n              :text \"Clojure\"}\n             {:msg :click :locator (shb/by-css-selector \"#search_button_homepage\")}\n             {:msg :wait-for :condition \n              (shb/presence-of-element-located \n               (shb/by-css-selector \".serp__results\"))}]]\n    (sh/send-message context scrape-result msg))\n  (sh/stop))\n----\n\nRunning `(example)` results in:\n\n[source,clojure]\n----\nuser\u003e (https://clojure.org/ https://en.wikipedia.org/wiki/Clojure https://github.com/clojure/clojure https://www.reddit.com/r/Clojure/ https://clojuredocs.org/ https://clojuredocs.org/clojure.core/when https://www.braveclojure.com/ https://repl.it/languages/clojure https://www.zhihu.com/question/21446061 https://leiningen.org/ https://clojure.github.io/clojure/ https://github.com/clojure https://www.tutorialspoint.com/clojure/clojure_basic_syntax.htm https://learnxinyminutes.com/docs/clojure/ https://cursive-ide.com/ https://clojurescript.org/ https://www.tutorialspoint.com/clojure/clojure_loops.htm http://www.clojurekoans.com/ https://www.braveclojure.com/clojure-for-the-brave-and-true/ https://marketplace.visualstudio.com/items?itemName=avli.clojure https://www.youtube.com/user/ClojureTV https://www.slant.co/options/1538/~clojure-review https://www.amazon.com/Clojure-Programming-Practical-Lisp-World/dp/1449394701 https://kimh.github.io/clojure-by-example/ https://ja.wikipedia.org/wiki/Clojure http://www.4clojure.\ncom/ https://developer.mozilla.org/en-US/docs/Web/JavaScript/Closures https://www.clojure.org/guides/getting_started https://en.wikibooks.org/wiki/Clojure_Programming)\n----\n\nException management\n~~~~~~~~~~~~~~~~~~~~\n\nBy default, Shelob prints out exceptions on standard output and continues the\ncrawling process; you can customise exception management by passing a custom\nexception management function to `send-message`, as such:\n\n[source,clojure]\n----\n\n....\n\n(defn exception-custom-fn\n  \"Prints out exception with a custom message\"\n  [_source e]\n  (println \"An exception occurred, this is a custom message -\" (.getMessage e)))\n\n(defn example\n  []\n  (sh/init context)\n  (let [msg [...]\n    (sh/send-message context scrape-result exception-custom-fn msg))\n  (sh/stop))\n----\n\nLicense\n~~~~~~~\n\nCopyright © 2019 7bridges s.r.l. — Distributed under the Apache License\n2.0.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F7bridges-eu%2Fshelob","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F7bridges-eu%2Fshelob","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F7bridges-eu%2Fshelob/lists"}