{"id":17296137,"url":"https://github.com/kirankotari/weblinks","last_synced_at":"2025-03-26T20:41:59.387Z","repository":{"id":42200764,"uuid":"475878867","full_name":"kirankotari/weblinks","owner":"kirankotari","description":"Weblinks, It get all links from a given website and we can apply filters on top of it to get desired results, when you are good you can start downloading them with -d flag.  In the library we are supporting plain webpages, authentication based webpages, proxy, authentication on proxy, etc.  We also support storing the config in either local/global configuration, for best practise we suggest not to store your password, you get a prompt on runtime where you can provide it.","archived":false,"fork":false,"pushed_at":"2023-03-09T15:54:36.000Z","size":42,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-02T08:16:50.824Z","etag":null,"topics":["auth","authentication","cli","command","fetch","filter","filterlinks","get","getlinks","links","proxy","python","web","weblinks"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kirankotari.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-03-30T12:55:58.000Z","updated_at":"2022-07-24T07:13:01.000Z","dependencies_parsed_at":"2022-07-17T06:00:36.102Z","dependency_job_id":null,"html_url":"https://github.com/kirankotari/weblinks","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kirankotari%2Fweblinks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kirankotari%2Fweblinks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kirankotari%2Fweblinks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kirankotari%2Fweblinks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kirankotari","download_url":"https://codeload.github.com/kirankotari/weblinks/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245733681,"owners_count":20663566,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["auth","authentication","cli","command","fetch","filter","filterlinks","get","getlinks","links","proxy","python","web","weblinks"],"created_at":"2024-10-15T11:12:05.256Z","updated_at":"2025-03-26T20:41:59.362Z","avatar_url":"https://github.com/kirankotari.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Weblinks\n\n[![License: Apache 2.0](https://img.shields.io/badge/License-Apache2-yellow.svg)](https://opensource.org/licenses/Apache-2.0)\n[![pytest](https://github.com/kirankotari/weblinks/actions/workflows/pytest.yml/badge.svg)](https://github.com/kirankotari/weblinks/actions/workflows/pytest.yml)\n![GitHub release (latest by date)](https://img.shields.io/github/v/release/kirankotari/weblinks)\n[![Downloads](https://static.pepy.tech/personalized-badge/weblinks?period=total\u0026units=international_system\u0026left_color=grey\u0026right_color=orange\u0026left_text=pypi%20downloads)](https://pepy.tech/project/weblinks)\n[![Downloads](https://static.pepy.tech/personalized-badge/weblinks?period=week\u0026units=international_system\u0026left_color=grey\u0026right_color=orange\u0026left_text=pypi%20downloads/week)](https://pepy.tech/project/weblinks)\n![GitHub all releases](https://img.shields.io/github/downloads/kirankotari/weblinks/total?label=github%20downloads)\n![GitHub top language](https://img.shields.io/github/languages/top/kirankotari/weblinks)\n![GitHub issues](https://img.shields.io/github/issues/kirankotari/weblinks)\n![GitHub pull requests](https://img.shields.io/github/issues-pr/kirankotari/weblinks)\n\n- [Introduction](#introduction)\n- [Pre-requisites](#pre-requisites)\n- [Installation and Downloads](#installation-and-downloads)\n- [Commands](#commands)\n- [Docs](#docs)\n- [Bug Tracker and Support](#bug-tracker-and-support)\n- [License and Copyright](#license-and-copyright)\n- [Author and Thanks](#author-and-thanks)\n\n\n## Introduction\n\nWeblinks, It get all links from a given website and we can apply filters on top of it to get desired results, when you are good you can start downloading them with `-d` flag.\n\nIn the library we are supporting plain webpages, authentication based webpages, proxy, authentication on proxy, etc. \n\nWe also support storing the config in either local/global configuration, for best practise we suggest not to store your password, you get a prompt on runtime wher you can provide it.\n\n## Pre-requisites\n\nA system need to support **curl commands** and **python3**\n\n## Installation and Downloads\n\n```shell\npip install weblinks\n```\n\n## Commands\n\n```shell\nweblinks --help\n```\n```\nusage: weblinks [-h] [-w WEB] [-s SUBSTRING] [-e EXT] [-d] [-u USERNAME]\n                [-p PASSWORD] [-g] [-l] [-v] [--proxy PROXY]\n                [--proxy-username PROXY_USERNAME]\n                [--proxy-password PROXY_PASSWORD] [--version]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -w WEB, --web WEB     the website\n  -s SUBSTRING, --substring SUBSTRING\n                        the sub-string in the links\n  -e EXT, --ext EXT     file extention\n  -d, --download        download links\n  -u USERNAME, --username USERNAME\n                        web login username\n  -p PASSWORD, --password PASSWORD\n                        web login password\n  -g, --global          global configuration\n  -l, --local           local configuration\n  -v, --verbosity\n  --proxy PROXY         proxy address\n  --proxy-username PROXY_USERNAME\n                        proxy username\n  --proxy-password PROXY_PASSWORD\n                        proxy password\n  --version             weblinks version\n```\n\n## Docs\n\n**Weblinks usage**\n\nTo see current lib. version\n```shell\nweblinks --version\n# weblinks version: 2.0\n```\n\nTo see python file from given url\n```shell\nweblinks --web https://www.python.org/ftp/python/3.8.13/ --substring Python\n# INFO     | 2022-07-23 16:23:33,603 | run     :117  | links found\n# Python-3.8.13.tar.xz\n# Python-3.8.13.tar.xz.asc\n# Python-3.8.13.tgz\n# Python-3.8.13.tgz.asc\n```\n\nStill wanted to filter add file extention\n```shell\nweblinks --web https://www.python.org/ftp/python/3.8.13/ --substring Python --ext .tgz\n# INFO     | 2022-07-23 16:23:33,603 | run     :117  | links found\n# Python-3.8.13.tgz\n```\n\nStart download, listed links are good\n```shell\nweblinks --web https://www.python.org/ftp/python/3.8.13/ --substring Python --ext .tgz -d\n# INFO     | 2022-07-23 16:25:34,807 | run     :117  | links found\n# Python-3.8.13.tgz\n# INFO     | 2022-07-23 16:25:34,807 | run     :124  | start download: Python-3.8.13.tgz\n# INFO     | 2022-07-23 16:25:34,807 | utils   :58   | downloading: Python-3.8.13.tgz\n# INFO     | 2022-07-23 16:25:36,849 | run     :126  | completed: Python-3.8.13.tgz\n```\n\nFor authentication\n```shell\nweblinks --web \u003curl\u003e --substring \u003csub\u003e --username \u003ckirankotari\u003e --password \u003cxxxxxx\u003e --ext .tgz\n# Note: don't add --password, it will ask dynamically\n```\n\nFor verbose add -v\n```shell\nweblinks --web \u003curl\u003e --substring \u003csub\u003e --username \u003ckirankotari\u003e --ext .tgz -v\n```\n\nTo store config add --local or --global respectively\n```shell\nweblinks --local --web \u003curl\u003e --username \u003ckirankotari\u003e --ext .tgz\n```\n\nFor proxy\n```shell\nweblinks --proxy \u003cip\u003e:\u003cport\u003e --web \u003curl\u003e --substring \u003csub\u003e\n```\n\nFor proxy with authentication\n```shell\nweblinks --proxy \u003cip\u003e:\u003cport\u003e --web \u003curl\u003e --substring \u003csub\u003e --proxy-username \u003cproxy user\u003e --proxy-password \u003cproxy password\u003e\n```\n\nAdding web authentication on top of proxy auth.\n```shell\nweblinks --proxy \u003cip\u003e:\u003cport\u003e --web \u003curl\u003e --substring \u003csub\u003e --proxy-username \u003cproxy user\u003e --proxy-password \u003cproxy password\u003e --username \u003cuser\u003e\n```\n\n## Bug Tracker and Support\n\n- Please report any suggestions, bug reports, or annoyances with weblinks through the [Github bug tracker](https://github.com/kirankotari/weblinks/issues). If you're having problems with general python issues, consider searching for a solution on [Stack Overflow](https://stackoverflow.com/search?q=).\n- If you can't find a solution for your problem or need more help, you can [ask a question](https://stackoverflow.com/questions/ask).\n\n## License and Copyright\n\n- weblinks is licensed [Apache 2.0](https://opensource.org/licenses/Apache-2.0) 2022\n\n## Author and Thanks\n\nWeblinks was developed by [Kiran Kumar Kotari](https://github.com/kirankotari)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkirankotari%2Fweblinks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkirankotari%2Fweblinks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkirankotari%2Fweblinks/lists"}