{"id":19929078,"url":"https://github.com/thescriptguy/generate-url-requests","last_synced_at":"2026-03-02T03:04:14.802Z","repository":{"id":203221010,"uuid":"709107502","full_name":"TheScriptGuy/generate-url-requests","owner":"TheScriptGuy","description":"Access a random sample of hosts from Umbrella's Top 1 Million hostnames","archived":false,"fork":false,"pushed_at":"2025-05-13T18:59:33.000Z","size":59,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-19T06:45:38.929Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TheScriptGuy.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-10-24T02:56:23.000Z","updated_at":"2025-05-13T18:59:36.000Z","dependencies_parsed_at":"2025-05-13T19:47:37.299Z","dependency_job_id":"d7dab224-0260-4ff0-a246-92d805bfa143","html_url":"https://github.com/TheScriptGuy/generate-url-requests","commit_stats":null,"previous_names":["thescriptguy/generate-url-requests"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/TheScriptGuy/generate-url-requests","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheScriptGuy%2Fgenerate-url-requests","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheScriptGuy%2Fgenerate-url-requests/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheScriptGuy%2Fgenerate-url-requests/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheScriptGuy%2Fgenerate-url-requests/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TheScriptGuy","download_url":"https://codeload.github.com/TheScriptGuy/generate-url-requests/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheScriptGuy%2Fgenerate-url-requests/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29991299,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-02T01:47:34.672Z","status":"online","status_checked_at":"2026-03-02T02:00:07.342Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T22:41:49.867Z","updated_at":"2026-03-02T03:04:14.779Z","avatar_url":"https://github.com/TheScriptGuy.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Multi-threaded URL Accessor\n\nThis script is a multi-threaded URL accessor. It downloads a list of top URLs from Umbrella (Top 1 million hosts), selects a random subset of them, and attempts to connect to each URL via HTTP and HTTPS protocols.\n\n## Features\n\n- Multi-threading to speed up the checking process\n- Handles both HTTPS (preferred) and HTTP protocols\n- Graceful termination upon receiving Ctrl+C\n- Allow insecure connections\n- Allow fixed delays or random delays for each worker thread.\n\n## Dependencies\n\n- `requests`\n\n## Installation\n\nBefore running the script, ensure that you have the `requests` library installed in your Python environment. You can install it using `pip`:\n\n```bash\npip install requests\n```\n\n## Usage\n\nYou can run this script from the command line using the following syntax:\n\n```bash\n$ python generate-requests.py [num_connections] [num_workers] [--insecure]\n```\n\n# Establishing 200 connections using 20 worker threads\n(as a side note here, unless you have a fairly resource friendly host to run this on, this could very quickly exhaust all resources on the server, or make too many DNS requests to your DNS providers.\n```bash\n$ python generate-requests.py 200 20\n```\n\n# To establish an insecure connection\n(To allow certificate warnings - useful for when you do not have the correct certificates installed.)\n```bash\n$ python generate-requests.py 200 20 --insecure\n```\n\n# Using a web proxy\nFirst edit the `generate-requests-proxy.py` file and adjust the proxy_settings variable:\n```python\n# Define a proxy setting\nproxy_settings = {\n    \"https\": \"http://proxy1.domain.com:8080\",\n    \"http\": \"http://proxy1.domain.com:8080\"\n}\n```\n\n```bash\n$ python generate-requests-proxy.py 200 20\n```\n\n# Configuring custom HTTP Headers\nEdit either the `generate-requests-proxy.py` or `generate-requests.py` and set http_headers appropriately.\n```python\n# Custom HTTP Headers\nhttp_headers = {\n    \"X-Authenticated-User\": \"\",\n    \"User-Agent\": \"MyTestUserAgent/1.0\"\n}\n```\n\n# Adding delays to requests\n## To add a fixed delay\nUse the `--delay` argument to indicate a fixed amount of time (in seconds) that a worker thread will pause before requesting a URL.\nIn this example, it'll establish 100 connections to URLs using 20 workers with a fixed 5 second delay between each request.\n```bash\n$ python generate-requests.py --delay 5 100 20\n```\n\n## To add a random delay\nUse the `--random-delay` argument to indicate a random amount of time (in seconds) that a worker thread will pause before requesting a URL.\nIn this example, it'll establish 100 connections to URLs using 20 workers with a random delay of up to 10 seconds.\n```bash\n$ python generate-requests.py --random-delay 10 100 20\n```\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthescriptguy%2Fgenerate-url-requests","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthescriptguy%2Fgenerate-url-requests","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthescriptguy%2Fgenerate-url-requests/lists"}