Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/carlospolop/malwareworld

System based on +500 blacklists to detect internet potencialy malicious hosts
https://github.com/carlospolop/malwareworld

blacklists malware-detection malwareworld proxies spam-detection tor

Last synced: about 1 month ago
JSON representation

System based on +500 blacklists to detect internet potencialy malicious hosts

Awesome Lists containing this project

README 

          
![MalwareWorld](docs/images/big_logo.png)



MalwareWorld aggregates public threat-intel blacklists into a static dataset (IPs, domains and IP ranges) and publishes it as a website on GitHub Pages.



- Website: `https://malwareworld.com/`

- HackTricks tools: `https://tools.hacktricks.wiki/`

- Blacklists used (URLs): `https://malwareworld.com/data/blacklists.txt`



## What this repo contains



- `docs/`: the GitHub Pages web UI (no backend).

- `scripts/`: a generator that downloads blacklists and produces:

  - textlists to download (`suspiciousIPs.txt`, `suspiciousDomains.txt`, `suspiciousRanges.txt`, …)

  - per-category downloads (`type__domains.txt`, `type__ips.txt`, `type__ranges.txt`)

  - sharded JSON for exact lookups from the UI (IP/domain/range)

  - maps + stats per category

  - monthly archive artifacts (optional; see below)



Generation uses SQLite on disk to keep memory usage low.



## GitHub Pages setup



1) Repo → Settings → Pages

2) Source: **GitHub Actions**

3) Run the workflow `.github/workflows/publish-release-assets.yml` once (or wait for the schedule).



The workflows publish a GitHub Release with the generated artifacts and also deploy the same artifacts under the Pages site at `/data/` so the UI can fetch them without CORS issues.



## Run locally (web UI + data generation)



1) Install:



`npm install`



2) Generate the site data:



`npm run generate:site`



Useful env vars:



`MW_LIMIT_BLACKLISTS=10 MW_CONCURRENCY=10 MW_OUTPUT_DIR=release-assets npm run generate:site`



For flaky sources (common on CI):



`MW_CONCURRENCY=12 MW_RETRY_COUNT=3 MW_RETRY_DELAY_MS=90000 npm run generate:site`



Monthly archive (union of all non-whitelist items seen during the month):



`MW_MONTHLY=1 MW_MONTHLY_DB_PATH=release-assets/monthly-YYYY-MM-archive.sqlite npm run generate:site`



Optional seed to merge the current run into an existing monthly archive:



`MW_MONTHLY_SEED_PATH=path/to/monthly-YYYY-MM-archive.sqlite`



3) Serve the UI:



`python3 -m http.server 4173 --directory docs`



Option A (recommended local dev): generate into `docs/data/` so the UI auto-detects it:



`MW_OUTPUT_DIR=docs/data npm run generate:site`



Open `http://127.0.0.1:4173/`



Option B: serve `release-assets/` separately and point the UI to it:



`python3 -m http.server 4174 --directory release-assets`



Open `http://127.0.0.1:4173/?releaseBase=http://127.0.0.1:4174/`



4) Quick end-to-end smoke test (generates a small subset and verifies outputs + HTTP fetches):



`npm test`



## CLI lookup tools (Node + Python)



The same sharded JSON layout used by the UI can be queried from the command line. Scripts live in `tools/lookup/`.



Local data (uses `docs/data/` if present):



`node tools/lookup/lookup.js example.com`



`python3 tools/lookup/lookup.py 1.1.1.1`



Remote release assets:



`node tools/lookup/lookup.js example.com --base https://github.com///releases/latest/download/`



`python3 tools/lookup/lookup.py 1.1.1.1 --base https://malwareworld.com/data/`



## Note about removals



This repo previously exposed a Node.js “library” API (including external intelligence lookups). That code path is removed: MalwareWorld is now focused on static data generation + GitHub Pages.



## Monthly archives (how it works)



MalwareWorld can keep a monthly union of all malicious IPs/domains/ranges/URLs with their categories and IP geolocation. When enabled:



- `scripts/generate-site-data.js` attaches/creates a monthly SQLite archive:

  `monthly-YYYY-MM-archive.sqlite`.

- Each run merges the current dataset into the archive (non‑whitelist only).

- It also generates monthly map and stats assets:

  `monthly-YYYY-MM-map_{Type}.geojson` and `monthly-YYYY-MM-stats_{Type}.json`.

- The workflow `.github/workflows/publish-release-assets.yml` publishes those files as a release

  and updates `docs/data/monthly/index.json` with the month → release URL mapping.



The UI reads `docs/data/monthly/index.json` and shows a Month selector in the Maps section.

When a month is selected, the UI loads the monthly map/stats files from the release URL.

Because GitHub Releases do not provide CORS headers, the UI fetches monthly files through a

configurable CORS proxy (default: `https://api.allorigins.win/raw?url=`). Override via:



`?corsProxy=https://your-proxy/?url=` or disable with `?corsProxy=none`.



## Downloadable files (from the web UI)



These files are published under `/data/` on GitHub Pages and also as release assets:



- `blacklists.txt` (all source blacklist URLs): [https://malwareworld.com/data/blacklists.txt](https://malwareworld.com/data/blacklists.txt)

- `blacklists.json` (run stats + blacklist metadata): [https://malwareworld.com/data/blacklists.json](https://malwareworld.com/data/blacklists.json)

- `suspiciousIPs.txt`: [https://malwareworld.com/data/suspiciousIPs.txt](https://malwareworld.com/data/suspiciousIPs.txt)

- `suspiciousDomains.txt`: [https://malwareworld.com/data/suspiciousDomains.txt](https://malwareworld.com/data/suspiciousDomains.txt)

- `suspiciousRanges.txt`: [https://malwareworld.com/data/suspiciousRanges.txt](https://malwareworld.com/data/suspiciousRanges.txt)

- `type_BadReputation_domains.txt`: [https://malwareworld.com/data/type_BadReputation_domains.txt](https://malwareworld.com/data/type_BadReputation_domains.txt)

- `type_BadReputation_ips.txt`: [https://malwareworld.com/data/type_BadReputation_ips.txt](https://malwareworld.com/data/type_BadReputation_ips.txt)

- `type_BadReputation_ranges.txt`: [https://malwareworld.com/data/type_BadReputation_ranges.txt](https://malwareworld.com/data/type_BadReputation_ranges.txt)

- `type_Malware_domains.txt`: [https://malwareworld.com/data/type_Malware_domains.txt](https://malwareworld.com/data/type_Malware_domains.txt)

- `type_Malware_ips.txt`: [https://malwareworld.com/data/type_Malware_ips.txt](https://malwareworld.com/data/type_Malware_ips.txt)

- `type_Malware_ranges.txt`: [https://malwareworld.com/data/type_Malware_ranges.txt](https://malwareworld.com/data/type_Malware_ranges.txt)

- `type_KnownAttacker_domains.txt`: [https://malwareworld.com/data/type_KnownAttacker_domains.txt](https://malwareworld.com/data/type_KnownAttacker_domains.txt)

- `type_KnownAttacker_ips.txt`: [https://malwareworld.com/data/type_KnownAttacker_ips.txt](https://malwareworld.com/data/type_KnownAttacker_ips.txt)

- `type_KnownAttacker_ranges.txt`: [https://malwareworld.com/data/type_KnownAttacker_ranges.txt](https://malwareworld.com/data/type_KnownAttacker_ranges.txt)

- `type_Spammer_domains.txt`: [https://malwareworld.com/data/type_Spammer_domains.txt](https://malwareworld.com/data/type_Spammer_domains.txt)

- `type_Spammer_ips.txt`: [https://malwareworld.com/data/type_Spammer_ips.txt](https://malwareworld.com/data/type_Spammer_ips.txt)

- `type_Spammer_ranges.txt`: [https://malwareworld.com/data/type_Spammer_ranges.txt](https://malwareworld.com/data/type_Spammer_ranges.txt)

- `type_Phishing_domains.txt`: [https://malwareworld.com/data/type_Phishing_domains.txt](https://malwareworld.com/data/type_Phishing_domains.txt)

- `type_Phishing_ips.txt`: [https://malwareworld.com/data/type_Phishing_ips.txt](https://malwareworld.com/data/type_Phishing_ips.txt)

- `type_Phishing_ranges.txt`: [https://malwareworld.com/data/type_Phishing_ranges.txt](https://malwareworld.com/data/type_Phishing_ranges.txt)

- `type_CryptoCurrencies_domains.txt`: [https://malwareworld.com/data/type_CryptoCurrencies_domains.txt](https://malwareworld.com/data/type_CryptoCurrencies_domains.txt)

- `type_CryptoCurrencies_ips.txt`: [https://malwareworld.com/data/type_CryptoCurrencies_ips.txt](https://malwareworld.com/data/type_CryptoCurrencies_ips.txt)

- `type_CryptoCurrencies_ranges.txt`: [https://malwareworld.com/data/type_CryptoCurrencies_ranges.txt](https://malwareworld.com/data/type_CryptoCurrencies_ranges.txt)

- `type_HideSource_domains.txt`: [https://malwareworld.com/data/type_HideSource_domains.txt](https://malwareworld.com/data/type_HideSource_domains.txt)

- `type_HideSource_ips.txt`: [https://malwareworld.com/data/type_HideSource_ips.txt](https://malwareworld.com/data/type_HideSource_ips.txt)

- `type_HideSource_ranges.txt`: [https://malwareworld.com/data/type_HideSource_ranges.txt](https://malwareworld.com/data/type_HideSource_ranges.txt)

- `type_Adware_domains.txt`: [https://malwareworld.com/data/type_Adware_domains.txt](https://malwareworld.com/data/type_Adware_domains.txt)

- `type_Adware_ips.txt`: [https://malwareworld.com/data/type_Adware_ips.txt](https://malwareworld.com/data/type_Adware_ips.txt)

- `type_Adware_ranges.txt`: [https://malwareworld.com/data/type_Adware_ranges.txt](https://malwareworld.com/data/type_Adware_ranges.txt)

- `type_DGA_domains.txt`: [https://malwareworld.com/data/type_DGA_domains.txt](https://malwareworld.com/data/type_DGA_domains.txt)

- `type_DGA_ips.txt`: [https://malwareworld.com/data/type_DGA_ips.txt](https://malwareworld.com/data/type_DGA_ips.txt)

- `type_DGA_ranges.txt`: [https://malwareworld.com/data/type_DGA_ranges.txt](https://malwareworld.com/data/type_DGA_ranges.txt)



Monthly archive artifacts (not in Pages; only in Releases):



- `monthly-YYYY-MM-archive.sqlite`: use the release base URL from [https://malwareworld.com/data/monthly/index.json](https://malwareworld.com/data/monthly/index.json)

- `monthly-YYYY-MM-map_.geojson`: use the release base URL from [https://malwareworld.com/data/monthly/index.json](https://malwareworld.com/data/monthly/index.json)

- `monthly-YYYY-MM-stats_.json`: use the release base URL from [https://malwareworld.com/data/monthly/index.json](https://malwareworld.com/data/monthly/index.json)



## Monthly archive workflow notes



- The Pages site always serves the latest dataset under `/data/`.

- Monthly archives and monthly map/stats are kept in Releases (not in Pages) to avoid

  unbounded Pages growth.

- The monthly index file (`docs/data/monthly/index.json`) is committed by the workflow so the

  UI can discover months without calling the GitHub API.



## License



MIT