{"id":23647638,"url":"https://github.com/ronin-rb/ronin-web-spider","last_synced_at":"2025-08-01T07:12:10.746Z","repository":{"id":64414670,"uuid":"451033948","full_name":"ronin-rb/ronin-web-spider","owner":"ronin-rb","description":"A collection of common web spidering routines","archived":false,"fork":false,"pushed_at":"2024-08-04T07:12:38.000Z","size":180,"stargazers_count":8,"open_issues_count":2,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-10-29T14:22:28.710Z","etag":null,"topics":["crawler","infosec","recon","ruby","scraper","spider","utils","web","websecurity"],"latest_commit_sha":null,"homepage":"https://ronin-rb.dev","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ronin-rb.png","metadata":{"files":{"readme":"README.md","changelog":"ChangeLog.md","contributing":null,"funding":null,"license":"COPYING.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"open_collective":"ronin-rb","patreon":"roninrb"}},"created_at":"2022-01-23T07:28:17.000Z","updated_at":"2024-08-04T07:12:41.000Z","dependencies_parsed_at":"2023-11-23T22:32:50.164Z","dependency_job_id":"d2e43ffa-11a8-46e6-a3a0-cb21afc231c2","html_url":"https://github.com/ronin-rb/ronin-web-spider","commit_stats":{"total_commits":154,"total_committers":1,"mean_commits":154.0,"dds":0.0,"last_synced_commit":"1231b045cb2071c29fdb18ab44cfc66de999ffa7"},"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ronin-rb%2Fronin-web-spider","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ronin-rb%2Fronin-web-spider/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ronin-rb%2Fronin-web-spider/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ronin-rb%2Fronin-web-spider/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ronin-rb","download_url":"https://codeload.github.com/ronin-rb/ronin-web-spider/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":231584942,"owners_count":18396154,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","infosec","recon","ruby","scraper","spider","utils","web","websecurity"],"created_at":"2024-12-28T14:28:54.864Z","updated_at":"2025-08-01T07:12:10.726Z","avatar_url":"https://github.com/ronin-rb.png","language":"Ruby","funding_links":["https://opencollective.com/ronin-rb","https://patreon.com/roninrb"],"categories":[],"sub_categories":[],"readme":"# ronin-web-spider\n\n[![CI](https://github.com/ronin-rb/ronin-web-spider/actions/workflows/ruby.yml/badge.svg)](https://github.com/ronin-rb/ronin-web-spider/actions/workflows/ruby.yml)\n[![Code Climate](https://codeclimate.com/github/ronin-rb/ronin-web-spider.svg)](https://codeclimate.com/github/ronin-rb/ronin-web-spider)\n[![Gem Version](https://badge.fury.io/rb/ronin-web-spider.svg)](https://badge.fury.io/rb/ronin-web-spider)\n\n* [Website](https://ronin-rb.dev/)\n* [Source](https://github.com/ronin-rb/ronin-web-spider)\n* [Issues](https://github.com/ronin-rb/ronin-web-spider/issues)\n* [Documentation](https://ronin-rb.dev/docs/ronin-web-spider/frames)\n* [Discord](https://discord.gg/6WAb3PsVX9) |\n  [Mastodon](https://infosec.exchange/@ronin_rb)\n\n## Description\n\nronin-web-spider is a collection of common web spidering routines using the\n[spidr] gem.\n\n## Features\n\n* Built on top of the battle tested and versatile [spidr] gem.\n* Provides additional callback methods:\n  * [every_host][docs-every_host] - yields every unique host name that's\n    spidered.\n  * [every_cert][docs-every_cert] - yields every unique SSL/TLS certificate\n    encountered while spidering.\n  * [every_favicon][docs-every_favicon] - yields every favicon file that's\n    encountered while spidering.\n  * [every_html_comment][docs-every_html_comment] - yields every HTML comment.\n  * [every_javascript][docs-every_javascript] - yields all JavaScript source\n    code from either inline `\u003cscript\u003e` or `.js` files.\n  * [every_javascript_string][docs-every_javascript_string] - yields every\n    single-quoted or double-quoted String literal from all JavaScript source\n    code.\n  * [every_javascript_relative_path_string][docs-every_javascript_relative_path_string] -\n    yields every relative path JavaScript string (ex: `foo/bar`).\n  * [every_javascript_absolute_path_string][docs-every_javascript_absolute_path_string] -\n    yields every relative path JavaScript string (ex: `/foo/bar`).\n  * [every_javascript_path_string][docs-every_javascript_path_string] -\n    yields every relative path JavaScript string (ex: `foo/bar` or `/foo/bar`).\n  * [every_javascript_url_string][docs-every_javascript_url_string] -\n    yields every URL JavaScript string (ex: `https://example.com/foo/bar`).\n  * [every_javascript_comment][docs-every_javascript_comment] - yields every\n    JavaScript comment.\n  * [every_comment][docs-every_comment] - yields every HTML or JavaScript\n    comment.\n* Supports archiving spidered pages to a directory or git repository.\n* Has 97% documentation coverage.\n* Has 94% test coverage.\n\n[docs-every_host]: https://ronin-rb.dev/docs/ronin-web-spider/Ronin/Web/Spider/Agent.html#every_host-instance_method\n[docs-every_cert]: https://ronin-rb.dev/docs/ronin-web-spider/Ronin/Web/Spider/Agent.html#every_cert-instance_method\n[docs-every_favicon]: https://ronin-rb.dev/docs/ronin-web-spider/Ronin/Web/Spider/Agent.html#every_favicon-instance_method\n[docs-every_html_comment]: https://ronin-rb.dev/docs/ronin-web-spider/Ronin/Web/Spider/Agent.html#every_html_comment-instance_method\n[docs-every_javascript]: https://ronin-rb.dev/docs/ronin-web-spider/Ronin/Web/Spider/Agent.html#every_javascript-instance_method\n[docs-every_javascript_string]: https://ronin-rb.dev/docs/ronin-web-spider/Ronin/Web/Spider/Agent.html#every_javascript_string-instance_method\n[docs-every_javascript_relative_path_string]: https://ronin-rb.dev/docs/ronin-web-spider/Ronin/Web/Spider/Agent.html#every_javascript_relative_path_string-instance_method\n[docs-every_javascript_absolute_path_string]: https://ronin-rb.dev/docs/ronin-web-spider/Ronin/Web/Spider/Agent.html#every_javascript_absolute_path_string-instance_method\n[docs-every_javascript_path_string]: https://ronin-rb.dev/docs/ronin-web-spider/Ronin/Web/Spider/Agent.html#every_javascript_path_string-instance_method\n[docs-every_javascript_url_string]: https://ronin-rb.dev/docs/ronin-web-spider/Ronin/Web/Spider/Agent.html#every_javascript_url_string-instance_method\n[docs-every_javascript_comment]: https://ronin-rb.dev/docs/ronin-web-spider/Ronin/Web/Spider/Agent.html#every_javascript_comment-instance_method\n[docs-every_comment]: https://ronin-rb.dev/docs/ronin-web-spider/Ronin/Web/Spider/Agent.html#every_comment-instance_method\n\n## Examples\n\nSpider a host:\n\n```ruby\nrequire 'ronin/web/spider'\n\nRonin::Web::Spider.start_at('http://tenderlovemaking.com/') do |agent|\n  # ...\nend\n```\n\nSpider a host:\n\n```ruby\nRonin::Web::Spider.host('solnic.eu') do |agent|\n  # ...\nend\n```\n\nSpider a domain (and any sub-domains):\n\n```ruby\nRonin::Web::Spider.domain('ruby-lang.org') do |agent|\n  # ...\nend\n```\n\nSpider a site:\n\n```ruby\nRonin::Web::Spider.site('http://www.rubyflow.com/') do |agent|\n  # ...\nend\n```\n\nSpider multiple hosts:\n\n```ruby\nRonin::Web::Spider.start_at('http://company.com/', hosts: ['company.com', /host[\\d]+\\.company\\.com/]) do |agent|\n  # ...\nend\n```\n\nDo not spider certain links:\n\n```ruby\nRonin::Web::Spider.site('http://company.com/', ignore_links: [%{^/blog/}]) do |agent|\n  # ...\nend\n```\n\nDo not spider links on certain ports:\n\n```ruby\nRonin::Web::Spider.site('http://company.com/', ignore_ports: [8000, 8010, 8080]) do |agent|\n  # ...\nend\n```\n\nDo not spider links blacklisted in robots.txt:\n\n```ruby\nRonin::Web::Spider.site('http://company.com/', robots: true) do |agent|\n  # ...\nend\n```\n\nPrint out visited URLs:\n\n```ruby\nRonin::Web::Spider.site('http://www.rubyinside.com/') do |spider|\n  spider.every_url { |url| puts url }\nend\n```\n\nBuild a URL map of a site:\n\n```ruby\nurl_map = Hash.new { |hash,key| hash[key] = [] }\n\nRonin::Web::Spider.site('http://intranet.com/') do |spider|\n  spider.every_link do |origin,dest|\n    url_map[dest] \u003c\u003c origin\n  end\nend\n```\n\nPrint out the URLs that could not be requested:\n\n```ruby\nRonin::Web::Spider.site('http://company.com/') do |spider|\n  spider.every_failed_url { |url| puts url }\nend\n```\n\nFinds all pages which have broken links:\n\n```ruby\nurl_map = Hash.new { |hash,key| hash[key] = [] }\n\nspider = Ronin::Web::Spider.site('http://intranet.com/') do |spider|\n  spider.every_link do |origin,dest|\n    url_map[dest] \u003c\u003c origin\n  end\nend\n\nspider.failures.each do |url|\n  puts \"Broken link #{url} found in:\"\n\n  url_map[url].each { |page| puts \"  #{page}\" }\nend\n```\n\nSearch HTML and XML pages:\n\n```ruby\nRonin::Web::Spider.site('http://company.com/') do |spider|\n  spider.every_page do |page|\n    puts \"\u003e\u003e\u003e #{page.url}\"\n\n    page.search('//meta').each do |meta|\n      name = (meta.attributes['name'] || meta.attributes['http-equiv'])\n      value = meta.attributes['content']\n\n      puts \"  #{name} = #{value}\"\n    end\n  end\nend\n```\n\nPrint out the titles from every page:\n\n```ruby\nRonin::Web::Spider.site('https://www.ruby-lang.org/') do |spider|\n  spider.every_html_page do |page|\n    puts page.title\n  end\nend\n```\n\nPrint out every HTTP redirect:\n\n```ruby\nRonin::Web::Spider.host('company.com') do |spider|\n  spider.every_redirect_page do |page|\n    puts \"#{page.url} -\u003e #{page.headers['Location']}\"\n  end\nend\n```\n\nFind what kinds of web servers a host is using, by accessing the headers:\n\n```ruby\nservers = Set[]\n\nRonin::Web::Spider.host('company.com') do |spider|\n  spider.all_headers do |headers|\n    servers \u003c\u003c headers['server']\n  end\nend\n```\n\nPause the spider on a forbidden page:\n\n```ruby\nRonin::Web::Spider.host('company.com') do |spider|\n  spider.every_forbidden_page do |page|\n    spider.pause!\n  end\nend\n```\n\nSkip the processing of a page:\n\n```ruby\nRonin::Web::Spider.host('company.com') do |spider|\n  spider.every_missing_page do |page|\n    spider.skip_page!\n  end\nend\n```\n\nSkip the processing of links:\n\n```ruby\nRonin::Web::Spider.host('company.com') do |spider|\n  spider.every_url do |url|\n    if url.path.split('/').find { |dir| dir.to_i \u003e 1000 }\n      spider.skip_link!\n    end\n  end\nend\n```\n\nDetect when a new host name is spidered:\n\n```ruby\nRonin::Web::Spider.domain('example.com') do |spider|\n  spider.every_host do |host|\n    puts \"Spidering #{host} ...\"\n  end\nend\n```\n\nDetect when a new SSL/TLS certificate is encountered:\n\n```ruby\nRonin::Web::Spider.domain('example.com') do |spider|\n  spider.every_cert do |cert|\n    puts \"Discovered new cert for #{cert.subject.command_name}, #{cert.subject_alt_name}\"\n  end\nend\n```\n\nPrint the MD5 checksum of every `favicon.ico` file:\n\n```ruby\nRonin::Web::Spider.domain('example.com') do |spider|\n  spider.every_favicon do |page|\n    puts \"#{page.url}: #{page.body.md5}\"\n  end\nend\n```\n\nPrint every HTML comment:\n\n```ruby\nRonin::Web::Spider.domain('example.com') do |spider|\n  spider.every_html_comment do |comment|\n    puts comment\n  end\nend\n```\n\nPrint all JavaScript source code:\n\n```ruby\nRonin::Web::Spider.domain('example.com') do |spider|\n  spider.every_javascript do |js|\n    puts js\n  end\nend\n```\n\nPrint every JavaScript string literal:\n\n```ruby\nRonin::Web::Spider.domain('example.com') do |spider|\n  spider.every_javascript_string do |str|\n    puts str\n  end\nend\n```\n\nPrint every JavaScript URL string literal:\n\n```ruby\nRonin::Web::Spider.domain('example.com') do |spider|\n  spider.every_javascript_url_string do |url|\n    puts url\n  end\nend\n```\n\nPrint every JavaScript comment:\n\n```ruby\nRonin::Web::Spider.domain('example.com') do |spider|\n  spider.every_javascript_comment do |comment|\n    puts comment\n  end\nend\n```\n\nPrint every HTML and JavaScript comment:\n\n```ruby\nRonin::Web::Spider.domain('example.com') do |spider|\n  spider.every_comment do |comment|\n    puts comment\n  end\nend\n```\n\nSpider a host and archive every web page:\n\n```ruby\nrequire 'ronin/web/spider'\nrequire 'ronin/web/spider/archive'\n\nRonin::Web::Spider::Archive.open('path/to/root') do |archive|\n  Ronin::Web::Spider.every_page(host: 'example.com') do |page|\n    archive.write(page.url,page.body)\n  end\nend\n```\n\nSpider a host and archive every web page to a Git repository:\n\n```ruby\nrequire 'ronin/web/spider/git_archive'\nrequire 'ronin/web/spider'\nrequire 'date'\n\nRonin::Web::Spider::GitArchive.open('path/to/root') do |archive|\n  archive.commit(\"Updated #{Date.today}\") do\n    Ronin::Web::Spider.every_page(host: 'example.com') do |page|\n      archive.write(page.url,page.body)\n    end\n  end\nend\n```\n\n## Requirements\n\n* [Ruby] \u003e= 3.0.0\n* [spidr] ~\u003e 0.7\n* [ronin-support] ~\u003e 1.0\n\n## Install\n\n```shell\n$ gem install ronin-web-spider\n```\n\n### Gemfile\n\n```ruby\ngem 'ronin-web-spider', '~\u003e 0.1'\n```\n\n### gemspec\n\n```ruby\ngem.add_dependency 'ronin-web-spider', '~\u003e 0.1'\n```\n\n## Development\n\n1. [Fork It!](https://github.com/ronin-rb/ronin-web-spider/fork)\n2. Clone It!\n3. `cd ronin-web-spider/`\n4. `bundle install`\n5. `git checkout -b my_feature`\n6. Code It!\n7. `bundle exec rake spec`\n8. `git push origin my_feature`\n\n## License\n\nCopyright (c) 2006-2025 Hal Brodigan (postmodern.mod3 at gmail.com)\n\nronin-web-spider is free software: you can redistribute it and/or modify\nit under the terms of the GNU Lesser General Public License as published\nby the Free Software Foundation, either version 3 of the License, or\n(at your option) any later version.\n\nronin-web-spider is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\nGNU Lesser General Public License for more details.\n\nYou should have received a copy of the GNU Lesser General Public License\nalong with ronin-web-spider.  If not, see \u003chttps://www.gnu.org/licenses/\u003e.\n\n[Ruby]: https://www.ruby-lang.org\n[spidr]: https://github.com/postmodern/spidr#readme\n[ronin-support]: https://github.com/ronin-rb/ronin-support#readme\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fronin-rb%2Fronin-web-spider","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fronin-rb%2Fronin-web-spider","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fronin-rb%2Fronin-web-spider/lists"}