{"id":14978922,"url":"https://github.com/thesis/shale","last_synced_at":"2025-12-12T01:11:38.662Z","repository":{"id":20602042,"uuid":"23883000","full_name":"thesis/shale","owner":"thesis","description":"A Clojure-backed replacement for Selenium hubs.","archived":false,"fork":false,"pushed_at":"2017-06-02T23:05:43.000Z","size":1236,"stargazers_count":14,"open_issues_count":29,"forks_count":2,"subscribers_count":28,"default_branch":"master","last_synced_at":"2024-09-29T02:05:56.708Z","etag":null,"topics":["clojure","selenium","selenium-webdriver","shale"],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thesis.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-09-10T16:52:16.000Z","updated_at":"2023-10-29T23:03:53.000Z","dependencies_parsed_at":"2022-08-31T08:41:46.644Z","dependency_job_id":null,"html_url":"https://github.com/thesis/shale","commit_stats":null,"previous_names":["cardforcoin/shale"],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thesis%2Fshale","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thesis%2Fshale/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thesis%2Fshale/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thesis%2Fshale/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thesis","download_url":"https://codeload.github.com/thesis/shale/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219859520,"owners_count":16556036,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clojure","selenium","selenium-webdriver","shale"],"created_at":"2024-09-24T13:58:38.204Z","updated_at":"2025-12-12T01:11:38.614Z","avatar_url":"https://github.com/thesis.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# shale\n\nShale, a Clojure-backed Selenium hub replacement\n\n[![Circle CI](https://circleci.com/gh/cardforcoin/shale/tree/master.png?style=badge)](https://circleci.com/gh/cardforcoin/shale/tree/master)\n\n[![Clojars Project](http://clojars.org/shale/latest-version.svg)](http://clojars.org/shale)\n\n## Progress\n\nShale is a lightweight replacement for a Selenium hub. We've found hubs to be\nmore trouble than they're worth operationally, but we still need some of the\nmanagement features.\n\nCurrently shale lets you get, create, and delete webdriver sessions, and\nmaintains \"reservation\" and \"tag\" metadata to make querying against sessions\neasy.\n\nNodes can be discovered via the AWS API, a list of URLs, or through a custom\n`INodePool` implementation. See the example config for details.\n\n## Running\n\nClone the repo\n\n```sh\ngit clone git@github.com:cardforcoin/shale.git\n```\n\nIn development, you can run the service with `lein`.\n\n```sh\nlein run\n# OR, run it in a REPL\nlein repl\n\u003e (init)\n\u003e (start)\nINFO [shale.configurer] Loaded shale config...\n{:node-list [\"http://localhost:4444/wd/hub\" \"http://localhost:4443/wd/hub\"],\n :node-max-sessions 6,\n :port 5000,\n :webdriver-timeout 3000,\n :start-webdriver-timeout 5000}\nINFO [shale.nodes] - Starting the node pool...\nINFO [shale.sessions] - Starting session pool...\nINFO [shale.core] - Starting Jetty...\n```\n\nTo deploy the service, make a jar and run that.\n\n```sh\nlein uberjar\njava -jar target/shale-0.3.0-SNAPSHOT-standalone.jar\n```\n\n## Configuration\n\nBy default, a config file- `config.clj`- is expected on the classpath.\n`example-config.clj` shows the various options. If you'd like to provide a full\npath to an alternative config file, set an environment variable `CONFIG_FILE`\nwith the path, or include the option using another method supported by\n[environ][environ].\n\nAll grid nodes should be configured so that they don't attempt to register with\na hub (`register:false` in the node config), and should be accessible via the\nmachine running shale.\n\n[environ]: https://github.com/weavejester/environ\n\n## Examples\n\nSession creation, selection, and modification uses an S-expression inspired\nsyntax.\n\n### curl\n\n#### Get or create a new session\n\n```bash\ncurl -d '{\"require\": [\"browser_name\", \"phantomjs\"]}' -XPOST \\\n     http://localhost:5000/sessions/ \\\n     --header \"Content-Type:application/json\"\n```\n\n```json\n{\n  \"id\": \"05e9229d-356b-46a3-beae-f8ab02cea7db\",\n  \"webdriver_id\": \"bd416414-e80f-4556-9d97-ebcfdc44425f\",\n  \"reserved\": false,\n  \"node\": {\"url\":\"http://localhost:5555/wd/hub\", \"id\":\"2c28f0f2-e479-4501-a05d-a0991793abd7\"},\n  \"browser_name\": \"phantomjs\",\n  \"tags\": []\n}\n```\n\n#### List all the active sessions\n\n```bash\ncurl http://localhost:5000/sessions/\n```\n\n```json\n[{\n  \"id\": \"05e9229d-356b-46a3-beae-f8ab02cea7db\",\n  \"webdriver_id\": \"bd416414-e80f-4556-9d97-ebcfdc44425f\",\n  \"reserved\": false,\n  \"node\": {\"url\":\"http://localhost:5555/wd/hub\", \"id\":\"2c28f0f2-e479-4501-a05d-a0991793abd7\"},\n  \"browser_name\": \"phantomjs\",\n  \"tags\": []\n}]\n```\n\n#### Create a new session with tags\n\n```bash\ncurl -d '{\"create\": {\"browser_name\":\"phantomjs\", \"tags\":[\"walmart\"]}}' \\\n     -XPOST http://localhost:5000/sessions/ \\\n     --header \"Content-Type:application/json\"\n```\n\n```json\n{\n  \"id\": \"05e9229d-356b-46a3-beae-f8ab02cea7db\",\n  \"webdriver_id\": \"bd416414-e80f-4556-9d97-ebcfdc44425f\",\n  \"reserved\": false,\n  \"node\": {\"url\":\"http://localhost:5555/wd/hub\", \"id\":\"2c28f0f2-e479-4501-a05d-a0991793abd7\"},\n  \"browser_name\": \"phantomjs\",\n  \"tags\": [\"walmart\"]\n}\n```\n\n#### Get or create a new reserved session with tags\n\n```bash\ncurl -d '{\"require\": [\"and [[\"browser_name\", \"phantomjs\"] [\"tag\", \"walmart\"]]],\n          \"modify\": [[\"reserve\", true]]}' \\\n     -XPOST http://localhost:5000/sessions/ \\\n     --header \"Content-Type:application/json\"\n```\n\n```json\n{\n  \"id\": \"05e9229d-356b-46a3-beae-f8ab02cea7db\",\n  \"webdriver_id\": \"bd416414-e80f-4556-9d97-ebcfdc44425f\",\n  \"reserved\": true,\n  \"node\": {\"url\":\"http://localhost:5555/wd/hub\", \"id\":\"2c28f0f2-e479-4501-a05d-a0991793abd7\"},\n  \"browser_name\": \"phantomjs\",\n  \"tags\": [\"walmart\"]\n}\n```\n\n#### Create a session on a particular node\n\n```bash\ncurl -d '{\"create\": {\"browser_name\":\"phantomjs\", \"node\":{\"id\": \"\u003cnode id\u003e\"}}}' \\\n     -XPOST http://localhost:5000/sessions/ \\\n     --header \"Content-Type:application/json\"\n```\n\n#### Create a Chrome session using a proxy (manually)\n\nYou can create sessions with custom options, like proxy settings, disabling\nlocal storage, or any other settings exposed by Selenium's\n`DesiredCapabilities`.\n\n```bash\ncurl -d '{\"create\": {\"browser_name\":\"chrome\", \"extra_desired_capabilities\": {\"chromeOptions\": {\"args\": [\"--proxy-server=socks5://\u003chost\u003e:\u003cport\u003e\", \"--disable-plugins\",\"--disable-local-storage\"]}}}' \\\n     -XPOST http://localhost:5000/sessions/ \\\n     --header \"Content-Type:application/json\"\n```\n\nNote, though, that there are better ways to manage session proxies!\n\n#### Unreserve a session and add a tag\n\nPATCH accepts an array of modifications, including `change_tag`, `reserve`, and\n`go_to_url`.\n\n```bash\ncurl -d '[[\"change_tag\", {\"action\": \"add\", \"tag\":\"walmart\"}], [\"reserve\", true]]' \\\n     -XPATCH http://localhost:5000/sessions/05e9229d-356b-46a3-beae-f8ab02cea7db \\\n     --header \"Content-Type:application/json\"\n```\n\n```json\n{\n  \"id\": \"05e9229d-356b-46a3-beae-f8ab02cea7db\",\n  \"webdriver_id\": \"bd416414-e80f-4556-9d97-ebcfdc44425f\",\n  \"reserved\": false,\n  \"node\": {\"url\":\"http://localhost:5555/wd/hub\", \"id\":\"2c28f0f2-e479-4501-a05d-a0991793abd7\"},\n  \"browser_name\": \"phantomjs\",\n  \"tags\": [\"walmart\", \"logged-in\"]\n}\n```\n\n#### Delete a session\n\nNote that this will de-allocate the Selenium driver.\n\n```bash\ncurl -XDELETE http://localhost:5000/sessions/05e9229d-356b-46a3-beae-f8ab02cea7db\n```\n\n```\ntrue\n```\n\n#### Delete all sessions\n\n```bash\ncurl -XDELETE http://localhost:5000/sessions/\n```\n\n```json\ntrue\n```\n\n### Proxies\n\nShale also includes its own proxy management that allows proxies to be shared\nbetween sessions.\n\nAn initial proxy list can be provided in the config.\n\n#### Add a new proxy\n\n```bash\ncurl -XPOST http://localhost:5000/proxies/ \\\n     -d '{\n            \"public_ip\": \"8.8.8.8\",\n            \"host\": \"127.0.0.1\",\n            \"port\": 1234,\n            \"type\":\"socks5\"\n            \"shared\":true\n         }' \\\n     --header \"Content-Type:application/json\"\n```\n\n```json\n{\n    \"id\":\"f7c64a2c-595d-434c-80f0-15c9751ddcc8\",\n    \"public_ip\":\"127.0.0.1\",\n    \"host\":\"128.0.0.1\",\n    \"port\":1234,\n    \"type\":\"socks5\",\n    \"active\":true,\n    \"shared\":true\n}\n```\nA proxy with `\"shared\": false` can only be used by sessions that reference it\nby ID. Those with `\"shared\": true` are candidates for use by other sessions.\n\nThe `public_ip` is an optional attribute to keep track of a proxies\npublic-facing IP. This can be useful if you're testing public sites on the\ninternet, and have an IP whitelist.\n\nCurrently, only SOCKS proxies are supported.\n\n#### List all proxies\n\n```bash\ncurl http://localhost:5000/proxies/\n```\n\n```json\n[\n    {\n        \"id\":\"f7c64a2c-595d-434c-80f0-15c9751ddcc8\",\n        \"public_ip\":\"8.8.8.8\",\n        \"host\":\"127.0.0.1\",\n        \"port\":1234,\n        \"type\":\"socks5\",\n        \"active\":true,\n        \"shared\":true\n    }\n]\n```\n\n#### Create a Chrome session with a new proxy\n\n```bash\ncurl -d '{\"create\": {\"browser_name\":\"chrome\", \"proxy\": {\"type\": \"socks5\", \"host\": \"\u003chost\u003e\", \"port\":\u003cport\u003e, \"shared\": true}}}'\n     -XPOST http://localhost:5000/sessions/ \\\n     --header \"Content-Type:application/json\"\n```\n\nIf the proxy hasn't been seen before, it will be recorded as a candidate for use\nby other sessions. To avoid that behavior, set `\"shared\": false`.\n\n#### Get or create a Chrome session with an existing proxy\n\n```bash\ncurl -d '{\"require\": [\"and\" [[\"browser_name\", \"chrome\"] [\"proxy\", [\"id\", \"f7c64a2c-595d-434c-80f0-15c9751ddcc8\"]]]]}'\n     -XPOST http://localhost:5000/sessions/ \\\n     --header \"Content-Type:application/json\"\n```\n\n### Clojure\n\nThe Clojure client returns functional web drivers using `clj-webdriver`,\nand includes a macro to make working with drivers easier.\n\nHere's an example of how to get-or-create, reserve, use, and release a driver\nusing the `with-webdriver*` macro, inspired by the `clj-webdriver` examples.\n\n```clojure\n;; Log into Github\n;;\n(use '[shale.client :only [with-driver]])\n(use 'clj-webdriver.taxi)\n(with-webdriver* {:browser-name :firefox :tags [\"github\"]}\n  (to \"https://github.com\")\n  (click \"a[href*='login']\")\n  (input-text \"#login_field\" \"your_username\")\n  (-\u003e \"#password\"\n    (input-text \"your_password\")\n    submit))\n```\n\nSee the [clj-webdriver docs][clj-webdriver] and the client source for more details.\n\n### Python\n\nThere is also a Python client with its [own examples and documentation][shale-python].\n\n[clj-webdriver]: http://semperos.github.io/clj-webdriver/\n\n[shale-python]: https://github.com/cardforcoin/shale-python\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthesis%2Fshale","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthesis%2Fshale","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthesis%2Fshale/lists"}