{"id":19929073,"url":"https://github.com/thescriptguy/molasses-masses","last_synced_at":"2025-05-03T10:31:31.583Z","repository":{"id":256055644,"uuid":"852605270","full_name":"TheScriptGuy/molasses-masses","owner":"TheScriptGuy","description":"A repostitory of hosting providers","archived":false,"fork":false,"pushed_at":"2025-04-29T16:56:28.000Z","size":31455,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-29T17:45:30.957Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TheScriptGuy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-09-05T05:17:31.000Z","updated_at":"2025-04-29T16:56:31.000Z","dependencies_parsed_at":"2024-11-23T15:18:13.476Z","dependency_job_id":"d303d8de-30f4-497e-b51b-a9a9496eb374","html_url":"https://github.com/TheScriptGuy/molasses-masses","commit_stats":null,"previous_names":["thescriptguy/molasses-masses"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheScriptGuy%2Fmolasses-masses","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheScriptGuy%2Fmolasses-masses/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheScriptGuy%2Fmolasses-masses/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheScriptGuy%2Fmolasses-masses/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TheScriptGuy","download_url":"https://codeload.github.com/TheScriptGuy/molasses-masses/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252178849,"owners_count":21707083,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T22:41:49.603Z","updated_at":"2025-05-03T10:31:31.578Z","avatar_url":"https://github.com/TheScriptGuy.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# :honey_pot: Project Molasses-Masses\n\n## :globe_with_meridians: Overview\n\nProject Molasses-Masses is designed to provide a dynamically updated list of hosting providers from around the world. This list is based on the prefixes advertised as part of the BGP Autonomous System numbers.\n\nAcknowledging that hosting providers themselves are not inherently problematic, some of their customers may have malicious intentions. Therefore, we adopt a \"block first\" approach, creating exceptions to allow legitimate traffic through as needed.\n\nThis project is currently tracking `13713` IPv4 Subnets and `2028` IPv6 Subnets.\n\n## :hammer_and_wrench: How It Works\n\nThe project uses a Python script (`molasses_masses.py`) to fetch and process IP prefixes associated with various hosting providers.\n\n### Input\n\n- The script reads from an input file named `hosting-providers-source.txt`.\n- This file should contain a list of tuples with hosting provider names and their AS numbers.\n\n### Output\n\nThe script generates output in the `data` directory:\n\n1. :page_facing_up: `combined-v4.txt`: A list of all IPv4 prefixes ([View on Cloudflare](https://files.broda.io/molasses_masses/combined-v4.txt))\n2. :page_facing_up: `combined-v6.txt`: A list of all IPv6 prefixes ([View on Cloudflare](https://files.broda.io/molasses_masses/combined-v6.txt))\n3. :notebook_with_decorative_cover: Source data for the day\n\nThese files are published to Cloudflare for easy access and distribution.\n\n## :rocket: Usage\n\nTo run the script:\n\n```bash\npython molasses_masses.py --if hosting-providers-source.txt --od data --verbose --combined --summarize\n```\n\n### Options:\n\n- `--if`: Input file (required)\n- `--od`: Output directory (required)\n- `--verbose`: Enable verbose logging\n- `--combined`: Create combined output files\n- `--summarize`: Summarize IP addresses\n\n# IP and Subnet Search Tool\n\nThis Python script searches for an exact match or the presence of IP addresses or subnets within files or directories containing a list of IPs/subnets. It's designed to help you quickly locate IP addresses or subnets from large datasets.\n\n## :arrow_forward: Usage\n\nTo use this script, you need to provide either a directory or a file to search, along with the IP addresses or subnets you want to search for.\n\n### Command-line Arguments\n\n- `--directory` : Directory to search for files containing IP addresses or subnets.\n- `--file` : Specific file to search.\n- `--search` : Comma-separated list of IP addresses and/or subnets to search for. **(Required)**\n- `--verbose` : Enable verbose output for detailed processing information.\n\n### Examples\n\n#### Example 1: Search a Directory\n\nTo search all files in the `data` directory for the IP address `192.100.0.0/24`, run:\n\n```bash\npython script.py --directory data --search 192.100.0.0/24\n```\n\n#### Example 2: Search a Specific File\n\nTo search a specific file, such as `data/2024-09-09/global-communications-net-AS12615-v4.txt`, for multiple IP addresses or subnets, run:\n```bash\npython script.py --file data/2024-09-09/global-communications-net-AS12615-v4.txt --search 192.100.1.0/24,100.0.0.0/8\n```\n\n#### Example 3: Search with Verbose Output\n\nTo search the data directory with verbose output, run:\n```bash\npython script.py --directory data --search 192.100.1.0/24 --verbose\n```\n\n#### :page_facing_up: Example Output\nWhen running the script with a search query, you might see output similar to the following:\n```text\nSearching directory: data\nSearching file: data/2024-09-09/global-communications-net-AS12615-v4.txt\nSkipping invalid entry on line 5: InvalidSubnet/33\ndata/2024-09-09/global-communications-net-AS12615-v4.txt: line 10: 192.100.1.0/24\ndata/2024-09-09/amanah-tech-AS32489-v4.txt: line 15: 100.0.0.0/8\n...\n```\n\n### :bulb: Important Notes\n\n    At least one of --directory or --file must be provided. The script requires at least one input source to search.\n    The --search argument is required and should contain a comma-separated list of valid IP addresses or subnets.\n    The script skips invalid IP addresses or subnets found in files and will display a message if the --verbose flag is enabled.\n\n### :wrench: Additional Information\n\n    If the --directory argument is provided, the script will recursively search through all files within the directory (excluding hidden files and directories).\n    If the --file argument is used, only that specific file will be searched.\n    The search is performed using Python's ipaddress module, which checks for both exact matches and subnet overlaps.\n\n\n# Add Hosting Provider Script :computer:\n\nThis Python script (`add_hp.py`) allows you to easily add new hosting provider entries to the `hosting-providers-source.txt` file. It's designed to maintain a consistent format and prevent duplicate entries.\n\n## :rocket: Features\n\n- Add new hosting providers with their AS (Autonomous System) numbers\n- Prevent duplicate AS number entries\n- Maintain consistent file format\n- Provide feedback on successful additions and total entries\n\n## :book: Usage\n\n1. :open_file_folder: Ensure that `add_hp.py` and `hosting-providers-source.txt` are in the same directory.\n\n2. :computer: Open a terminal or command prompt and navigate to the directory containing the script.\n\n3. :keyboard: Run the script using the following command format:\n\n   ```\n   python add_hp.py \u003cname\u003e \u003cas_number\u003e\n   ```\n\n   Replace `\u003cname\u003e` with the hosting provider's name and `\u003cas_number\u003e` with their AS number.\n\n   Example:\n   ```\n   python add_hp.py example-host 12345\n   ```\n\n4. :white_check_mark: If successful, you'll see a confirmation message and the total number of entries in the file.\n\n## :warning: Input Validation\n\n- The `name` can contain letters, numbers, hyphens, and underscores.\n- The `as_number` must be a positive integer.\n\n## :x: Error Handling\n\nThe script will display error messages for the following situations:\n- Incorrect number of arguments\n- Invalid name format\n- Invalid AS number\n- Duplicate AS number in the existing file\n\n## :file_folder: File Format\n\nThe `hosting-providers-source.txt` file maintains the following format:\n\n```\n[\n(\"provider1\", 12345),\n(\"provider2\", 67890),\n...\n(\"last-provider\", 99999)\n]\n```\n\n## :warning: Disclaimer\n\nThis project is not meant to imply that all traffic from hosting providers is malicious. It's a tool for those who prefer a cautious approach to network traffic, allowing for precise control and whitelisting as needed. If you leverage VPS services, then it is up to you to filter out your applicable IP addresses/subnets from the list. I cannot be held liable for any actions of this block list.\n\n## :fire: Shorewall Blocklist\n\nI have created a bash script `molasses-masses-shorewall.sh` that will help download the files and update shorewall rules to reflect the list of subnets that should be blocked. It should only update the list of iptables if the files hosted on Cloudflare have been updated.\n\n## :handshake: Contributing\n\nContributions to Project Molasses-Masses are welcome! Please feel free to submit pull requests or open issues to improve the project.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthescriptguy%2Fmolasses-masses","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthescriptguy%2Fmolasses-masses","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthescriptguy%2Fmolasses-masses/lists"}