{"id":16347897,"url":"https://github.com/victoriadrake/hydra-link-checker","last_synced_at":"2025-03-16T15:31:13.596Z","repository":{"id":40651248,"uuid":"238715835","full_name":"victoriadrake/hydra-link-checker","owner":"victoriadrake","description":"Hydra: a multithreaded site-crawling link checker in Python standard library","archived":false,"fork":false,"pushed_at":"2023-02-20T14:32:51.000Z","size":31,"stargazers_count":122,"open_issues_count":3,"forks_count":26,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-10-12T00:46:48.731Z","etag":null,"topics":["ci-cd","link-checker","link-checking","python","website"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/victoriadrake.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-02-06T15:07:54.000Z","updated_at":"2024-10-07T14:15:31.000Z","dependencies_parsed_at":"2022-08-10T00:00:37.408Z","dependency_job_id":null,"html_url":"https://github.com/victoriadrake/hydra-link-checker","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victoriadrake%2Fhydra-link-checker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victoriadrake%2Fhydra-link-checker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victoriadrake%2Fhydra-link-checker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victoriadrake%2Fhydra-link-checker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/victoriadrake","download_url":"https://codeload.github.com/victoriadrake/hydra-link-checker/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221665308,"owners_count":16860214,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ci-cd","link-checker","link-checking","python","website"],"created_at":"2024-10-11T00:46:53.928Z","updated_at":"2024-10-27T10:49:46.021Z","avatar_url":"https://github.com/victoriadrake.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Hydra: multithreaded site-crawling link checker in Python\n\n![Tests status badge](https://github.com/victoriadrake/hydra-link-checker/workflows/test/badge.svg)\n\nA Python program that ~~crawls~~ slithers 🐍 a website for links and prints a YAML report of broken links.\n\n## Requires\n\nPython 3.6 or higher.\n\nThere are no external dependencies, Neo.\n\n## Usage\n\n```sh\n$ python hydra.py -h\nusage: hydra.py [-h] [--config CONFIG] URL\n```\n\nPositional arguments:\n\n- `URL`: The URL of the website to crawl. Ensure `URL` is absolute including schema, e.g. `https://example.com`.\n\nOptional arguments:\n\n- `-h`, `--help`: Show help message and exit\n- `--config CONFIG`, `-c CONFIG`: Path to a configuration file\n\nA broken links report will be output to stdout, so you may like to redirect this to a file.\n\nThe report will be [YAML](https://yaml.org/) formatted. To save the output to a file, run:\n\n```sh\npython hydra.py [URL] \u003e [PATH/TO/FILE.yaml]\n```\n\nYou can add the current date to the filename using a command substitution, such as:\n\n```sh\npython hydra.py [URL] \u003e /path/to/$(date '+%Y_%m_%d')_report.yaml\n```\n\nTo see how long Hydra takes to check your site, add `time`:\n\n```sh\ntime python hydra.py [URL]\n```\n\n### GitHub Action\n\nYou can easily incorporate Hydra as part of an automated process using the [link-snitch](https://github.com/victoriadrake/link-snitch) action.\n\n## Configuration\n\nHydra can accept an optional JSON configuration file for specific parameters, for example:\n\n```json\n{\n    \"OK\": [\n        200,\n        999,\n        403\n    ],\n    \"attrs\": [\n        \"href\"\n    ],\n    \"exclude_scheme_prefixes\": [\n        \"tel\"\n    ],\n    \"tags\": [\n        \"a\",\n        \"img\"\n    ],\n    \"threads\": 25,\n    \"timeout\": 30,\n    \"graceful_exit\": \"True\"\n}\n```\n\nTo use a configuration file, supply the filename:\n\n```sh\npython hydra.py https://example.com --config ./hydra-config.json\n```\n\nPossible settings:\n\n- `OK` - [HTTP response codes](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status) to consider as a successful link check. Defaults to `[200, 999]`.\n- `attrs` - Attributes of the HTML tags to check for links. Defaults to `[\"href\", \"src\"]`.\n- `exclude_scheme_prefixes` - HTTP scheme prefixes to exclude from checking. Defaults to `[\"tel:\", \"javascript:\"]`.\n- `tags` - HTML tags to check for links. Defaults to `[\"a\", \"link\", \"img\", \"script\"]`.\n- `threads` - Maximum workers to run. Defaults to `50`.\n- `timeout` - Maximum seconds to wait for HTTP response. Defaults to `60`.\n- `graceful_exit` - If set to `True`, and there are broken links present return `exit code 0` else return `exit code 1`.\n\n## Test\n\nRun:\n\n```sh\npython -m unittest tests/test.py\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvictoriadrake%2Fhydra-link-checker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvictoriadrake%2Fhydra-link-checker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvictoriadrake%2Fhydra-link-checker/lists"}