{"id":13910918,"url":"https://github.com/michaeltelford/broken_link_finder","last_synced_at":"2025-07-18T10:31:55.864Z","repository":{"id":45992074,"uuid":"78279060","full_name":"michaeltelford/broken_link_finder","owner":"michaeltelford","description":"Finds a websites broken links and reports back to you with a summary","archived":false,"fork":false,"pushed_at":"2024-10-30T12:37:06.000Z","size":145,"stargazers_count":6,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-08T14:43:55.680Z","etag":null,"topics":["broken-link-finder","broken-links","links","ruby","website","wgit"],"latest_commit_sha":null,"homepage":"https://rubygems.org/gems/broken_link_finder","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/michaeltelford.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-01-07T12:57:42.000Z","updated_at":"2024-10-30T12:37:09.000Z","dependencies_parsed_at":"2024-08-07T02:06:08.836Z","dependency_job_id":null,"html_url":"https://github.com/michaeltelford/broken_link_finder","commit_stats":null,"previous_names":["michaeltelford/broken-link-finder"],"tags_count":22,"template":false,"template_full_name":null,"purl":"pkg:github/michaeltelford/broken_link_finder","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaeltelford%2Fbroken_link_finder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaeltelford%2Fbroken_link_finder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaeltelford%2Fbroken_link_finder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaeltelford%2Fbroken_link_finder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/michaeltelford","download_url":"https://codeload.github.com/michaeltelford/broken_link_finder/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michaeltelford%2Fbroken_link_finder/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265742291,"owners_count":23820823,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["broken-link-finder","broken-links","links","ruby","website","wgit"],"created_at":"2024-08-07T00:01:50.167Z","updated_at":"2025-07-18T10:31:55.590Z","avatar_url":"https://github.com/michaeltelford.png","language":"Ruby","funding_links":[],"categories":["Ruby"],"sub_categories":[],"readme":"# Broken Link Finder\n\nDoes what it says on the tin - finds a website's broken links.\n\nSimply point it at a website and it will crawl all of its webpages searching for and identifing broken links. You will then be presented with a concise summary of any broken links found.\n\nBroken Link Finder is multi-threaded and uses `libcurl` under the hood, it's fast!\n\n## How It Works\n\nAny HTML element within `\u003cbody\u003e` with a `href` or `src` attribute is considered a link (this is [configurable](#Link-Extraction) however).\n\nFor each link on a given page, any of the following conditions constitutes that the link is broken:\n\n- An empty HTML response body is returned.\n- A response status code of `404 Not Found` is returned.\n- The HTML response body doesn't contain an element ID matching that of the link's fragment e.g. `http://server.com#about` must contain an element with `id=\"about\"` or the link is considered broken.\n- The link redirects more than 5 times consecutively.\n\n**Note**: Not all link types are supported.\n\nIn a nutshell, only HTTP(S) based links can be successfully verified by `broken_link_finder`. As a result some links on a page might be (recorded and) ignored. You should verify these links yourself manually. Examples of unsupported link types include `tel:*`, `mailto:*`, `ftp://*` etc.\n\nSee the [usage](#Usage) section below on how to check which links have been ignored during a crawl.\n\nWith that said, the usual array of HTTP URL features are supported including anchors/fragments, query strings and IRI's (non ASCII based URL's).\n\n## Made Possible By\n\n`broken_link_finder` relies heavily on the `wgit` Ruby gem by the same author. See its [repository](https://github.com/michaeltelford/wgit) for more details.\n\n## Installation\n\nOnly MRI Ruby is tested and supported, but `broken_link_finder` may work with other Ruby implementations.\n\nCurrently, the required MRI Ruby version is:\n\n`ruby '\u003e= 2.6', '\u003c 4'`\n\n### Using Bundler\n\n    $ bundle add broken_link_finder\n\n### Using RubyGems\n\n    $ gem install broken_link_finder\n\n### Verify\n\n    $ broken_link_finder version\n\n## Usage\n\nYou can check for broken links via the executable or library.\n\n### Executable\n\nInstalling this gem installs the `broken_link_finder` executable into your `$PATH`. The executable allows you to find broken links from your command line. For example:\n\n    $ broken_link_finder crawl http://txti.es\n\nAdding the `--recursive` flag would crawl the entire `txti.es` site, not just its index page.\n\nSee the [output](#Output) section below for an example of a site with broken links.\n\nYou can peruse all of the available executable flags with:\n\n    $ broken_link_finder help crawl\n\n### Library\n\nBelow is a simple script which crawls a website and outputs its broken links to `STDOUT`:\n\n\u003e main.rb\n\n```ruby\nrequire 'broken_link_finder'\n\nfinder = BrokenLinkFinder.new\nfinder.crawl_site 'http://txti.es' # Or use Finder#crawl_page for a single webpage.\nfinder.report                      # Or use Finder#broken_links and Finder#ignored_links\n                                   # for direct access to the link Hashes.\n```\n\nThen execute the script with:\n\n    $ ruby main.rb\n\nSee the full source code documentation [here](https://www.rubydoc.info/gems/broken_link_finder).\n\n## Output\n\nIf broken links are found then the output will look something like:\n\n```text\nCrawled http://txti.es\n7 page(s) containing 32 unique link(s) in 6.82 seconds\n\nFound 6 unique broken link(s) across 2 page(s):\n\nThe following broken links were found on 'http://txti.es/about':\nhttp://twitter.com/thebarrytone\n/doesntexist\nhttp://twitter.com/nwbld\ntwitter.com/txties\n\nThe following broken links were found on 'http://txti.es/how':\nhttp://en.wikipedia.org/wiki/Markdown\nhttp://imgur.com\n\nIgnored 3 unique unsupported link(s) across 2 page(s), which you should check manually:\n\nThe following links were ignored on 'http://txti.es':\ntel:+13174562564\nmailto:big.jim@jmail.com\n\nThe following links were ignored on 'http://txti.es/contact':\nftp://server.com\n```\n\nYou can provide the `--html` flag if you'd prefer a HTML based report.\n\n## Link Extraction\n\nYou can customise the XPath used to extract links from each crawled page. This can be done via the executable or library.\n\n### Executable\n\nAdd the `--xpath` (or `-x`) flag to the crawl command e.g.\n\n    $ broken_link_finder crawl http://txti.es -x //img/@src\n\n### Library\n\nSet the desired XPath using the accessor methods provided:\n\n\u003e main.rb\n\n```ruby\nrequire 'broken_link_finder'\n\n# Set your desired xpath before crawling...\nBrokenLinkFinder::link_xpath = '//img/@src'\n\n# Now crawl as normal and only your custom targeted links will be checked.\nBrokenLinkFinder.new.crawl_page 'http://txti.es'\n\n# Go back to using the default provided xpath as needed.\nBrokenLinkFinder::link_xpath = BrokenLinkFinder::DEFAULT_LINK_XPATH\n```\n\n## Contributing\n\nBug reports and feature requests are welcome on [GitHub](https://github.com/michaeltelford/broken-link-finder). Just raise an issue.\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).\n\n## Development\n\nAfter checking out the repo, run `bin/setup` to install dependencies. Then, run `bundle exec rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.\n\nTo install this gem onto your local machine, run `bundle exec rake install`.\n\nTo release a new gem version:\n- Update the deps in the `*.gemspec`, if necessary.\n- Update the version number in `version.rb` and add the new version to the `CHANGELOG`.\n- Run `bundle install`.\n- Run `bundle exec rake test` ensuring all tests pass.\n- Run `bundle exec rake compile` ensuring no warnings.\n- Run `bundle exec rake install \u0026\u0026 rbenv rehash`.\n- Manually test the executable.\n- Commit any changes and merge your dev branch into `master`. Push `master` to `origin`.\n- Run `bundle exec rake release[origin]`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichaeltelford%2Fbroken_link_finder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmichaeltelford%2Fbroken_link_finder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichaeltelford%2Fbroken_link_finder/lists"}