{"id":32176338,"url":"https://github.com/urlstechie/urlchecker-python","last_synced_at":"2025-10-21T19:58:20.662Z","repository":{"id":37782906,"uuid":"248832251","full_name":"urlstechie/urlchecker-python","owner":"urlstechie","description":":snake: :link: Python module and client for checking URLs","archived":false,"fork":false,"pushed_at":"2024-10-10T10:42:02.000Z","size":41028,"stargazers_count":21,"open_issues_count":17,"forks_count":13,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-10-21T19:58:11.469Z","etag":null,"topics":["hacktoberfest","python","url-check","url-checker","urlchecker"],"latest_commit_sha":null,"homepage":"https://urlchecker-python.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/urlstechie.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["vsoch","SuperKogito"]}},"created_at":"2020-03-20T19:05:28.000Z","updated_at":"2025-04-01T16:05:45.000Z","dependencies_parsed_at":"2024-02-03T22:25:34.687Z","dependency_job_id":"a3c13c1f-7cb3-41f4-a7d9-2ea6c706041e","html_url":"https://github.com/urlstechie/urlchecker-python","commit_stats":{"total_commits":184,"total_committers":8,"mean_commits":23.0,"dds":0.3804347826086957,"last_synced_commit":"7dbd7ac171cf85788728b4cf5576c191f13c8399"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"purl":"pkg:github/urlstechie/urlchecker-python","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/urlstechie%2Furlchecker-python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/urlstechie%2Furlchecker-python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/urlstechie%2Furlchecker-python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/urlstechie%2Furlchecker-python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/urlstechie","download_url":"https://codeload.github.com/urlstechie/urlchecker-python/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/urlstechie%2Furlchecker-python/sbom","scorecard":{"id":912039,"data":{"date":"2025-08-11","repo":{"name":"github.com/urlstechie/urlchecker-python","commit":"11dd54a23f788c9a301da23799c3d979e9235046"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":4.7,"checks":[{"name":"Code-Review","score":6,"reason":"Found 15/25 approved changesets -- score normalized to 6","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/main.yml:1","Warn: no topLevel permission defined: .github/workflows/test.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/main.yml:18: update your workflow using https://app.stepsecurity.io/secureworkflow/urlstechie/urlchecker-python/main.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/test.yml:14: update your workflow using https://app.stepsecurity.io/secureworkflow/urlstechie/urlchecker-python/test.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/test.yml:31: update your workflow using https://app.stepsecurity.io/secureworkflow/urlstechie/urlchecker-python/test.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/test.yml:48: update your workflow using https://app.stepsecurity.io/secureworkflow/urlstechie/urlchecker-python/test.yml/master?enable=pin","Warn: containerImage not pinned by hash: Dockerfile:1: pin your Docker image by updating bitnami/minideb:buster to bitnami/minideb:buster@sha256:837cf5cdc9fdb8c6cc144b41e3626b28384697f661862dc29a6c95e9e521e118","Warn: downloadThenRun not pinned by hash: Dockerfile:8-16","Warn: pipCommand not pinned by hash: Dockerfile:24-28","Warn: pipCommand not pinned by hash: Dockerfile:24-28","Warn: pipCommand not pinned by hash: .github/workflows/test.yml:23","Warn: pipCommand not pinned by hash: .github/workflows/test.yml:40","Warn: pipCommand not pinned by hash: .github/workflows/test.yml:41","Warn: pipCommand not pinned by hash: .github/workflows/test.yml:63","Warn: pipCommand not pinned by hash: .github/workflows/test.yml:64","Info:   0 out of   4 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   1 downloadThenRun dependencies pinned","Info:   0 out of   7 pipCommand dependencies pinned","Info:   0 out of   1 containerImage dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Packaging","score":10,"reason":"packaging workflow detected","details":["Info: Project packages its releases by way of GitHub Actions.: .github/workflows/main.yml:13"],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":9,"reason":"1 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: PYSEC-2022-238 / GHSA-h3qr-fjhm-jphw"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 27 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-24T19:34:10.822Z","repository_id":37782906,"created_at":"2025-08-24T19:34:10.822Z","updated_at":"2025-08-24T19:34:10.822Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280325299,"owners_count":26311419,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-21T02:00:06.614Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hacktoberfest","python","url-check","url-checker","urlchecker"],"created_at":"2025-10-21T19:58:19.734Z","updated_at":"2025-10-21T19:58:20.652Z","avatar_url":"https://github.com/urlstechie.png","language":"Python","funding_links":["https://github.com/sponsors/vsoch","https://github.com/sponsors/SuperKogito"],"categories":["List of Content"],"sub_categories":["Software"],"readme":"\u003cdiv style=\"text-align:center\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/urlstechie/urlchecker-python/master/docs/urlstechie.png\"/\u003e\u003c/div\u003e\n\n[![Build Status](https://travis-ci.com/urlstechie/urlchecker-python.svg?branch=master)](https://travis-ci.com/urlstechie/urlchecker-python) [![Documentation Status](https://readthedocs.org/projects/urlchecker-python/badge/?version=latest)](https://urlchecker-python.readthedocs.io/en/latest/?badge=latest) [![codecov](https://codecov.io/gh/urlstechie/urlchecker-python/branch/master/graph/badge.svg)](https://codecov.io/gh/urlstechie/urlchecker-python) [![Python](https://img.shields.io/badge/python-3.5%20%7C%203.6%20%7C%203.7-blue)](https://www.python.org/doc/versions/) [![CodeFactor](https://www.codefactor.io/repository/github/urlstechie/urlchecker-python/badge)](https://www.codefactor.io/repository/github/urlstechie/urlchecker-python) ![PyPI](https://img.shields.io/pypi/v/urlchecker) [![Downloads](https://pepy.tech/badge/urlchecker)](https://pepy.tech/project/urlchecker) [![License](https://img.shields.io/badge/license-MIT-brightgreen)](https://github.com/urlstechie/urlchecker-python/blob/master/LICENSE)\n\n\n# urlchecker-python\n\nThis is a python module to collect urls over static files (code and documentation)\nand then test for and report broken links. If you are interesting in using\nthis as a GitHub action, see [urlchecker-action](https://github.com/urlstechie/urlchecker-action). There are also container\nbases available on [quay.io/urlstechie/urlchecker](https://quay.io/repository/urlstechie/urlchecker?tab=tags). As of version\n0.0.26, we use multiprocessing so the checks run a lot faster, and you can set `URLCHECKER_WORKERS` to change the number of workers\n(defaults to 9). If you don't want multiprocessing, use version 0.0.25 or earlier.\n\n## Module Documentation\n\nA detailed documentation of the code is available under [urlchecker-python.readthedocs.io](https://urlchecker-python.readthedocs.io/en/latest/)\n\n## Usage\n\n### Install\n\nYou can install the urlchecker from [pypi](https://pypi.org/project/urlchecker).\nBefore you do, it's recommended to install fake-useragent from:\n\n```bash\npip install git+https://github.com/danger89/fake-useragent.git\n```\n\nAnd then urlchecker:\n\n```bash\n$ pip install urlchecker\n```\n\nor install from the repository directly:\n\n```bash\n$ git clone https://github.com/urlstechie/urlchecker-python.git\n$ cd urlchecker-python\n$ python setup.py install\n```\n\nInstallation will place a binary, `urlchecker` in your Python path.\n\n```bash\n$ which urlchecker\n/home/vanessa/anaconda3/bin/urlchecker\n```\n\n\n### Check Local Folder\n\nYour most likely use case will be to check a local directory with static files (documentation or code)\nfor files. In this case, you can use urlchecker check:\n\n```bash\n$ urlchecker check --help\n```\n```console\nusage: urlchecker check [-h] [-b BRANCH] [--subfolder SUBFOLDER] [--cleanup] [--serial] [--no-check-certs]\n                        [--force-pass] [--no-print] [--verbose] [--file-types FILE_TYPES] [--files FILES]\n                        [--exclude-urls EXCLUDE_URLS] [--exclude-patterns EXCLUDE_PATTERNS]\n                        [--exclude-files EXCLUDE_FILES] [--save SAVE] [--retry-count RETRY_COUNT] [--timeout TIMEOUT]\n                        path\n\npositional arguments:\n  path                  the local path or GitHub repository to clone and check\n\noptions:\n  -h, --help            show this help message and exit\n  -b BRANCH, --branch BRANCH\n                        if cloning, specify a branch to use (defaults to main)\n  --subfolder SUBFOLDER\n                        relative subfolder path within path (if not specified, we use root)\n  --cleanup             remove root folder after checking (defaults to False, no cleaup)\n  --serial              run checks in serial (no multiprocess)\n  --no-check-certs      Allow urls to validate that fail certificate checks\n  --force-pass          force successful pass (return code 0) regardless of result\n  --no-print            Skip printing results to the screen (defaults to printing to console).\n  --verbose             Print file names for failed urls in addition to the urls.\n  --file-types FILE_TYPES\n                        comma separated list of file extensions to check (defaults to .md,.py)\n  --files FILES         comma separated list of exact files or patterns to check.\n  --exclude-urls EXCLUDE_URLS\n                        comma separated links to exclude (no spaces)\n  --exclude-patterns EXCLUDE_PATTERNS\n                        comma separated list of patterns to exclude (no spaces)\n  --exclude-files EXCLUDE_FILES\n                        comma separated list of files and patterns to exclude (no spaces)\n  --save SAVE           Path to a csv file to save results to.\n  --retry-count RETRY_COUNT\n                        retry count upon failure (defaults to 2, one retry).\n  --timeout TIMEOUT     timeout (seconds) to provide to the requests library (defaults to 5)\n```\n\nYou have a lot of flexibility to define patterns of urls or files to skip,\nalong with the number of retries or timeout (seconds). The most basic usage will\ncheck an entire directory. Let's clone and check the urlchecker action:\n\n```bash\n$ git clone https://github.com/urlstechie/urlchecker-action.git\n$ cd urchecker-action\n```\n\nand run the simplest command to check the present working directory (.).\n\n```bash\n$ urlchecker check .\n           original path: .\n              final path: /tmp/urlchecker-action\n               subfolder: None\n                  branch: master\n                 cleanup: False\n              file types: ['.md', '.py']\n                   files: []\n               print all: True\n           urls excluded: []\n   url patterns excluded: []\n  file patterns excluded: []\n              force pass: False\n             retry count: 2\n                    save: None\n                 timeout: 5\n\n /tmp/urlchecker-action/README.md \n --------------------------------\nhttps://github.com/urlstechie/urlchecker-action/blob/master/LICENSE\nhttps://github.com/r-hub/docs/blob/bc1eac71206f7cb96ca00148dcf3b46c6d25ada4/.github/workflows/pr.yml\nhttps://img.shields.io/static/v1?label=Marketplace\u0026message=urlchecker-action\u0026color=blue?style=flat\u0026logo=github\nhttps://github.com/rseng/awesome-rseng\nhttps://github.com/rseng/awesome-rseng/blob/5f5cb78f8392cf10aec2f3952b305ae9611029c2/.github/workflows/urlchecker.yml\nhttps://github.com/HPC-buildtest/buildtest-framework/actions?query=workflow%3A%22Check+URLs%22\nhttps://www.codefactor.io/repository/github/urlstechie/urlchecker-action/badge\nhttps://github.com/berlin-hack-and-tell/berlinhackandtell.rocks\nhttps://github.com/urlstechie/urlchecker-action/issues\nhttps://github.com/USRSE/usrse.github.io\nhttps://github.com/berlin-hack-and-tell/berlinhackandtell.rocks/actions?query=workflow%3ACommands\nhttps://github.com/USRSE/usrse.github.io/blob/abcbed5f5703e0d46edb9e8850eea8bb623e3c1c/.github/workflows/urlchecker.yml\nhttps://github.com/urlstechie/urlchecker-action/releases\nhttps://img.shields.io/badge/license-MIT-brightgreen\nhttps://github.com/r-hub/docs/actions?query=workflow%3ACommands\nhttps://github.com/rseng/awesome-rseng/actions?query=workflow%3AURLChecker\nhttps://github.com/buildtesters/buildtest\nhttps://github.com/r-hub/docs\nhttps://www.codefactor.io/repository/github/urlstechie/urlchecker-action\nhttps://github.com/urlstechie/URLs-checker-test-repo\nhttps://github.com/marketplace/actions/urlchecker-action\nhttps://github.com/actions/checkout\nhttps://github.com/SuperKogito/URLs-checker/issues/1,https://github.com/SuperKogito/URLs-checker/issues/2\nhttps://github.com/SuperKogito/URLs-checker/issues/1,https://github.com/SuperKogito/URLs-checker/issues/2\nhttps://github.com/USRSE/usrse.github.io/actions?query=workflow%3A%22Check+URLs%22\nhttps://github.com/SuperKogito/Voice-based-gender-recognition/issues\nhttps://github.com/buildtesters/buildtest/blob/v0.9.1/.github/workflows/urlchecker.yml\nhttps://github.com/berlin-hack-and-tell/berlinhackandtell.rocks/blob/master/.github/workflows/urlchecker-pr-label.yml\n\n /tmp/urlchecker-action/examples/README.md \n -----------------------------------------\nhttps://github.com/urlstechie/urlchecker-action/releases\nhttps://github.com/urlstechie/urlchecker-action/issues\nhttps://help.github.com/en/actions/reference/events-that-trigger-workflows\n\n\nDone. The following urls did not pass:\nhttps://github.com/SuperKogito/URLs-checker/issues/1,https://github.com/SuperKogito/URLs-checker/issues/2\n```\n\nThe url that didn't pass above is an example parameter for the library! Let's add\na simple pattern to exclude it.\n\n```bash\n$ urlchecker check --exclude-pattern SuperKogito .\n           original path: .\n              final path: /tmp/urlchecker-action\n               subfolder: None\n                  branch: master\n                 cleanup: False\n              file types: ['.md', '.py']\n                   files: []\n               print all: True\n           urls excluded: []\n   url patterns excluded: ['SuperKogito']\n  file patterns excluded: []\n              force pass: False\n             retry count: 2\n                    save: None\n                 timeout: 5\n\n /tmp/urlchecker-action/README.md \n --------------------------------\nhttps://github.com/urlstechie/urlchecker-action/blob/master/LICENSE\nhttps://github.com/urlstechie/urlchecker-action/issues\nhttps://github.com/rseng/awesome-rseng/actions?query=workflow%3AURLChecker\nhttps://github.com/USRSE/usrse.github.io/actions?query=workflow%3A%22Check+URLs%22\nhttps://github.com/actions/checkout\nhttps://github.com/USRSE/usrse.github.io/blob/abcbed5f5703e0d46edb9e8850eea8bb623e3c1c/.github/workflows/urlchecker.yml\nhttps://github.com/r-hub/docs/blob/bc1eac71206f7cb96ca00148dcf3b46c6d25ada4/.github/workflows/pr.yml\nhttps://github.com/berlin-hack-and-tell/berlinhackandtell.rocks/blob/master/.github/workflows/urlchecker-pr-label.yml\nhttps://github.com/rseng/awesome-rseng\nhttps://www.codefactor.io/repository/github/urlstechie/urlchecker-action/badge\nhttps://github.com/urlstechie/URLs-checker-test-repo\nhttps://www.codefactor.io/repository/github/urlstechie/urlchecker-action\nhttps://github.com/r-hub/docs\nhttps://github.com/berlin-hack-and-tell/berlinhackandtell.rocks\nhttps://github.com/buildtesters/buildtest\nhttps://img.shields.io/badge/license-MIT-brightgreen\nhttps://github.com/urlstechie/urlchecker-action/releases\nhttps://github.com/marketplace/actions/urlchecker-action\nhttps://img.shields.io/static/v1?label=Marketplace\u0026message=urlchecker-action\u0026color=blue?style=flat\u0026logo=github\nhttps://github.com/r-hub/docs/actions?query=workflow%3ACommands\nhttps://github.com/HPC-buildtest/buildtest-framework/actions?query=workflow%3A%22Check+URLs%22\nhttps://github.com/buildtesters/buildtest/blob/v0.9.1/.github/workflows/urlchecker.yml\nhttps://github.com/berlin-hack-and-tell/berlinhackandtell.rocks/actions?query=workflow%3ACommands\nhttps://github.com/USRSE/usrse.github.io\nhttps://github.com/rseng/awesome-rseng/blob/5f5cb78f8392cf10aec2f3952b305ae9611029c2/.github/workflows/urlchecker.yml\n\n /tmp/urlchecker-action/examples/README.md \n -----------------------------------------\nhttps://help.github.com/en/actions/reference/events-that-trigger-workflows\nhttps://github.com/urlstechie/urlchecker-action/issues\nhttps://github.com/urlstechie/urlchecker-action/releases\n\n\nDone. All URLS passed.\n```\n\nWe can also filter by file types. If we want to do this (for example, to only check different file\ntypes) we might do any of the following:\n\n```bash\n# Check only html files\nurlchecker check --file-types *.html .\n\n# Check hidden flies\nurlchecker check --file-types \".*\" .\n\n# Check hidden files and html files\nurlchecker check --file-types \".*,*.html\" .\n```\n\n**Note that while some patterns will work without quotes, it's recommended for most**\nto use them because if the shell expands any part of the pattern, it will not work as\nexpected. By default, the urlchecker checks python and markdown. If a multiprocessing workers has an error,\nyou can also add `--serial` to run in serial and test. The run will be slower, but it's useful for debugging.\n\n```bash\n$ urlchecker check . --files \"content/docs/hacking/contributing/documentation/index.md\" --serial\n```\n\n### Check GitHub Repository\n\nBut wouldn't it be easier to not have to clone the repository first?\nOf course! We can specify a GitHub url instead, and add `--cleanup`\nif we want to clean up the folder after.\n\n```bash\n$ urlchecker check https://github.com/SuperKogito/SuperKogito.github.io.git\n```\n\nIf you specify any arguments for a white list (or any kind of expected list) make\nsure that you provide a comma separated list *without any spaces*\n\n```bash\n$ urlchecker check --exclude-files=README.md,_config.yml\n```\n\n### Save Results\n\nIf you want to save your results to file, perhaps for some kind of record or\nother data analysis, you can provide the `--save` argument:\n\n```bash\n$ urlchecker check --save results.csv .\n```\n\nThe file that you save to will include a comma separated value tabular listing\nof the urls, and their result. The result options are \"passed\" and \"failed\"\nand the default header is `URL,RESULT`. All of these defaults are exposed\nif you want to change them (e.g., using a tab separator or a different header)\nif you call the function from within Python. Here is an example of the default file\nproduced, which should satisfy most use cases:\n\n```\nURL,RESULT\nhttps://github.com/SuperKogito,passed\nhttps://www.google.com/,passed\nhttps://github.com/SuperKogito/Voice-based-gender-recognition/issues,passed\nhttps://github.com/SuperKogito/Voice-based-gender-recognition,passed\nhttps://github.com/SuperKogito/spafe/issues/4,passed\nhttps://github.com/SuperKogito/Voice-based-gender-recognition/issues/2,passed\nhttps://github.com/SuperKogito/spafe/issues/5,passed\nhttps://github.com/SuperKogito/URLs-checker/blob/master/README.md,passed\nhttps://img.shields.io/,passed\nhttps://github.com/SuperKogito/spafe/,passed\nhttps://github.com/SuperKogito/spafe/issues/3,passed\nhttps://www.google.com/,passed\nhttps://github.com/SuperKogito,passed\nhttps://github.com/SuperKogito/spafe/issues/8,passed\nhttps://github.com/SuperKogito/spafe/issues/7,passed\nhttps://github.com/SuperKogito/Voice-based-gender-recognition/issues/1,passed\nhttps://github.com/SuperKogito/spafe/issues,passed\nhttps://github.com/SuperKogito/URLs-checker/issues,passed\nhttps://github.com/SuperKogito/spafe/issues/2,passed\nhttps://github.com/SuperKogito/URLs-checker,passed\nhttps://github.com/SuperKogito/spafe/issues/6,passed\nhttps://github.com/SuperKogito/spafe/issues/1,passed\nhttps://github.com/SuperKogito/URLs-checker/README.md,failed\nhttps://github.com/SuperKogito/URLs-checker/issues/3,failed\nhttps://none.html,failed\nhttps://github.com/SuperKogito/URLs-checker/issues/2,failed\nhttps://github.com/SuperKogito/URLs-checker/README.md,failed\nhttps://github.com/SuperKogito/URLs-checker/issues/1,failed\nhttps://github.com/SuperKogito/URLs-checker/issues/4,failed\n```\n\n\n### Usage from Python\n\n#### Checking a Path\n\nIf you want to check a list of urls outside of the provided client, this is fairly easy to do!\nLet's say we have a path, our present working directory, and we want to check\n.py and .md files (the default)\n\n```python\nfrom urlchecker.core.check import UrlChecker\nimport os\n\npath = os.getcwd()\nchecker = UrlChecker(path)    \n# UrlChecker:/home/vanessa/Desktop/Code/urlstechie/urlchecker-python\n```\n\nAnd of course you can provide more substantial arguments to derive the original\nfile list:\n\n```python\nchecker = UrlChecker(\n    path=path,\n    file_types=[\".md\", \".py\", \".rst\"],\n    include_patterns=[],\n    exclude_files=[\"README.md\", \"LICENSE\"],\n    print_all=True,\n)\n```\nI can then run the checker like this:\n\n```python\nchecker.run()\n```\n\nOr with more customization of excluded urls:\n\n```python\nchecker.run(\n    exclude_urls=exclude_urls,\n    exclude_patterns=exclude_patterns,\n    retry_count=3,\n    timeout=5,\n)\n```\n\nYou'll get the results object returned, which is also available at `checker.results`,\na simple dictionary with \"passed\" and \"failed\" keys to show passes and fails across\nall files.\n\n```python\n{'passed': ['https://github.com/SuperKogito/spafe/issues/4',\n  'http://shachi.org/resources',\n  'https://superkogito.github.io/blog/SpectralLeakageWindowing.html',\n  'https://superkogito.github.io/figures/fig4.html',\n  'https://github.com/urlstechie/urlchecker-test-repo',\n  'https://www.google.com/',\n  ...\n  'https://github.com/SuperKogito',\n  'https://img.shields.io/',\n  'https://www.google.com/',\n  'https://docs.python.org/2'],\n 'failed': ['https://github.com/urlstechie/urlschecker-python/tree/master',\n  'https://github.com/SuperKogito/Voice-based-gender-recognition,passed',\n  'https://github.com/SuperKogito/URLs-checker/README.md',\n   ...\n  'https://superkogito.github.io/tables',\n  'https://github.com/SuperKogito/URLs-checker/issues/2',\n  'https://github.com/SuperKogito/URLs-checker/README.md',\n  'https://github.com/SuperKogito/URLs-checker/issues/4',\n  'https://github.com/SuperKogito/URLs-checker/issues/3',\n  'https://github.com/SuperKogito/URLs-checker/issues/1',\n  'https://none.html']}\n```\n\nYou can look at `checker.checks`, which is a dictionary of result objects,\norganized by the filename:\n\n```python\nfor file_name, result in checker.checks.items(): \n    print() \n    print(result) \n    print(\"Total Results: %s \" % result.count) \n    print(\"Total Failed: %s\" % len(result.failed)) \n    print(\"Total Passed: %s\" % len(result.passed)) \n\n...\n\nUrlCheck:/home/vanessa/Desktop/Code/urlstechie/urlchecker-python/tests/test_files/sample_test_file.md\nTotal Results: 26 \nTotal Failed: 6\nTotal Passed: 20\n\nUrlCheck:/home/vanessa/Desktop/Code/urlstechie/urlchecker-python/.pytest_cache/README.md\nTotal Results: 1 \nTotal Failed: 0\nTotal Passed: 1\n\nUrlCheck:/home/vanessa/Desktop/Code/urlstechie/urlchecker-python/.eggs/pytest_runner-5.2-py3.7.egg/ptr.py\nTotal Results: 0 \nTotal Failed: 0\nTotal Passed: 0\n\nUrlCheck:/home/vanessa/Desktop/Code/urlstechie/urlchecker-python/docs/source/conf.py\nTotal Results: 3 \nTotal Failed: 0\nTotal Passed: 3\n```\n\nFor any result object, you can print the list of passed, falied, white listed,\nor all the urls.\n\n```python\nresult.all                                                                                                                                                                       \n['https://www.sphinx-doc.org/en/master/usage/configuration.html',\n 'https://docs.python.org/3',\n 'https://docs.python.org/2']\n\nresult.failed                                                                                                                                                                    \n[]\n\nresult.exclude\n[]\n\nresult.passed                                                                                                                                                                    \n['https://www.sphinx-doc.org/en/master/usage/configuration.html',\n 'https://docs.python.org/3',\n 'https://docs.python.org/2']\n\nresult.count\n3\n```\n\n\n#### Checking a List of URls\n\nIf you start with a list of urls you want to check, you can do that too!\n\n```python\nfrom urlchecker.core.urlproc import UrlCheckResult\n\nurls = ['https://www.github.com', \"https://github.com\", \"https://banana-pudding-doesnt-exist.com\"]\n\n# Instantiate an empty checker to extract urls\nchecker = UrlCheckResult()\nFile name None is undefined or does not exist, skipping extraction.\n```\n\nIf you provied a file name, the urls would be extracted for you.\n\n```python\nchecker = UrlCheckResult(\n    file_name=file_name,\n    exclude_patterns=exclude_patterns,\n    exclude_urls=exclude_urls,\n    print_all=self.print_all,\n)\n```\n\nor you can provide all the parameters without the filename:\n\n```python\nchecker = UrlCheckResult(\n    exclude_patterns=exclude_patterns,\n    exclude_urls=exclude_urls,\n    print_all=self.print_all,\n)\n```\n\nIf you don't provide the file_name to check urls, you can give the urls\nyou defined previously directly to the `check_urls` function:\n\n\n```python\nchecker.check_urls(urls)\n\nhttps://www.github.com\nhttps://github.com\nHTTPSConnectionPool(host='banana-pudding-doesnt-exist.com', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('\u003curllib3.connection.VerifiedHTTPSConnection object at 0x7f989abdfa10\u003e: Failed to establish a new connection: [Errno -2] Name or service not known'))\nhttps://banana-pudding-doesnt-exist.com\n```\n\nAnd of course you can specify a timeout and retry:\n\n```python\nchecker.check_urls(urls, retry_count=retry_count, timeout=timeout)\n```\n\nAfter you run the checker you can get all the urls, the passed,\nand failed sets:\n\n```python\nchecker.failed                                                                                                                                                                   \n['https://banana-pudding-doesnt-exist.com']\n\nchecker.passed                                                                                                                                                                   \n['https://www.github.com', 'https://github.com']\n\nchecker.all                                                                                                                                                                      \n['https://www.github.com',\n 'https://github.com',\n 'https://banana-pudding-doesnt-exist.com']\n\nchecker.all                                                                                                                                                                      \n['https://www.github.com',\n 'https://github.com',\n 'https://banana-pudding-doesnt-exist.com']\n\nchecker.count                                                                                                                                                                    \n3\n```\n\nIf you have any questions, please don't hesitate to [open an issue](https://github.com/urlstechie/urlchecker-python).\n\n### Docker\n\nA Docker container is provided if you want to build a base container with urlchecker,\nmeaning that you don't need to install it on your host. You can build the container as\nfollows:\n\n```bash\ndocker build -t urlchecker .\n```\n\nAnd then the entrypoint will expose the urlchecker.\n\n```bash\ndocker run -it urlschecker\n```\n\n## Development\n\n### Organization\n\nThe module is organized as follows:\n\n```\n├── client              # command line client\n├── main                # functions for supported integrations (e.g., GitHub)\n├── core                # core file and url processing tools\n└── version.py          # package and versioning\n```\n\nIn the \"client\" folder, for example, the commands that are exposed for the client\n(e.g., check) would named accordingly, e.g., `client/check.py`.\nFunctions for Github are be provided in `main/github.py`. This organization should\nbe fairly straight forward to always find what you are looking for.\n\n### Drivers\n\nTo test more difficult urls, we use a web driver, and you can choose between:\n\n - [Chrome Driver](https://chromedriver.chromium.org/downloads)\n - [Gecko Driver](https://github.com/mozilla/geckodriver/releases) (firefox)\n \nboth to be used with selenium. This driver is optional, but will come by default with our action. To install\nit, you can download the driver at either of the links above and ensure you install selenium:\n\n```bash\n$ pip install urlchecker[selenium]\n```\nand either:\n\n1. Add it directly to your path\n2. Export the directory where it lives as `URLCHECKER_DRIVERS_PATH`\n3. Put it in the root of the urlchecker clone (it will be looked for here)\n\n\n## Support\n\nIf you need help, or want to suggest a project for the organization,\nplease [open an issue](https://github.com/urlstechie/urlchecker-python)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Furlstechie%2Furlchecker-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Furlstechie%2Furlchecker-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Furlstechie%2Furlchecker-python/lists"}