https://github.com/ivan-sincek/domain-extractor
Extract valid or partially valid domain names and IPs from malicious or invalid URLs.
https://github.com/ivan-sincek/domain-extractor
bug-bounty computer-forensics defensive-security domain domain-name ethical-hacking extractor incident-response ip penetration-testing python red-team-engagement security threat-hunting threat-intelligence url
Last synced: 8 months ago
JSON representation
Extract valid or partially valid domain names and IPs from malicious or invalid URLs.
- Host: GitHub
- URL: https://github.com/ivan-sincek/domain-extractor
- Owner: ivan-sincek
- License: mit
- Archived: true
- Created: 2020-08-08T12:49:57.000Z (about 5 years ago)
- Default Branch: master
- Last Pushed: 2023-06-19T21:10:00.000Z (over 2 years ago)
- Last Synced: 2025-02-16T14:56:51.655Z (8 months ago)
- Topics: bug-bounty, computer-forensics, defensive-security, domain, domain-name, ethical-hacking, extractor, incident-response, ip, penetration-testing, python, red-team-engagement, security, threat-hunting, threat-intelligence, url
- Language: Python
- Homepage:
- Size: 4.88 KB
- Stars: 8
- Watchers: 2
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Domain Extractor
Extract valid or partially valid domain names and IPs from malicious or invalid URLs.
Keep in mind that the algorithm is not that perfect, there might be false positives.
Tested on Kali Linux v2023.1 (64-bit).
Check the testing URLs [here](https://github.com/ivan-sincek/domain-extractor/blob/master/examples/urls.txt) and the results [here](https://github.com/ivan-sincek/domain-extractor/blob/master/examples/results.json).
Made for educational purposes. I hope it will help!
Future plans:
* detect IPv6 addresses.
## How to Run
Open your preferred console from [/src/](https://github.com/ivan-sincek/domain-extractor/tree/master/src) and run the commands shown below.
Install required packages:
```fundamental
pip3 install -r requirements.txt
```
Run the script:
```fundamental
python3 domain_extractor.py
```
## Extract Results
Extract hosts from the results:
```bash
jq -r '.[].hosts[]' results.json | sort -u -f | tee -a hosts.txt
```
Extract URLs with valid or partially valid hosts from the results:
```bash
jq -r '.[] | if (.hosts != []) then (.original) else (empty) end' results.json | sort -u -f | tee -a valid_urls.txt
```
Extract URLs with no valid nor partially valid hosts from the results:
```bash
jq -r '.[] | if (.hosts == []) then (.original) else (empty) end' results.json | sort -u -f | tee -a invalid_urls.txt
```
## Usage
```fundamental
Domain Extractor v3.0 ( github.com/ivan-sincek/domain-extractor )
Usage: python3 domain_extractor.py -f file -o out
Example: python3 domain_extractor.py -f malicious_urls.txt -o results.json
DESCRIPTION
Extract valid or partially valid domain names and IPs from URLs
FILE
File with URLs you want to extract data from
-f - malicious_urls.txt | etc.
OUT
Output file
-o - results.json | etc.
```