{"id":27121432,"url":"https://github.com/snowkluster/tap","last_synced_at":"2026-04-11T19:04:13.646Z","repository":{"id":286106296,"uuid":"855681919","full_name":"snowkluster/TAP","owner":"snowkluster","description":"Threat Analysis Platform - A Data collection and normalisation platform to keep with threat actors","archived":false,"fork":false,"pushed_at":"2025-05-11T16:09:19.000Z","size":110294,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-11T16:28:56.987Z","etag":null,"topics":["docker","fastapi","playwright","postgresql","tor","web-scraper"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/snowkluster.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-09-11T09:33:03.000Z","updated_at":"2025-05-11T16:09:22.000Z","dependencies_parsed_at":"2025-04-30T07:29:58.193Z","dependency_job_id":null,"html_url":"https://github.com/snowkluster/TAP","commit_stats":null,"previous_names":["snowkluster/tap"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/snowkluster/TAP","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snowkluster%2FTAP","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snowkluster%2FTAP/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snowkluster%2FTAP/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snowkluster%2FTAP/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/snowkluster","download_url":"https://codeload.github.com/snowkluster/TAP/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snowkluster%2FTAP/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278322146,"owners_count":25967874,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-04T02:00:05.491Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","fastapi","playwright","postgresql","tor","web-scraper"],"created_at":"2025-04-07T10:58:56.197Z","updated_at":"2025-10-04T14:11:37.427Z","avatar_url":"https://github.com/snowkluster.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Threat Analysis Platform (TAP)\n\n![GitLeaks](https://img.shields.io/badge/protected%20by-gitleaks-blue)\n[![License](https://img.shields.io/github/license/snowkluster/tap)](./LICENSE)\n![Version](https://img.shields.io/badge/version-1.0.0-blue)\n![Build Status](https://img.shields.io/github/actions/workflow/status/snowkluster/TAP/main.yml)\n![Docker](https://img.shields.io/badge/docker-ready-brightgreen)\n![Python](https://img.shields.io/badge/python-3.11%2B-blue)\n![Go](https://img.shields.io/badge/go-1.21%2B-blue)\n![Node.js](https://img.shields.io/badge/node.js-22%2B-green)\n![NGINX](https://img.shields.io/badge/nginx-latest-green)\n![PostgreSQL](https://img.shields.io/badge/postgresql-latest-blue)\n\nTAP provides analyst access to multiple cyber crime forums in a single *TAP* allowing them to streamline their threat intelligence and data collection capabities, TAP provides access to multiple cyber crime forums, ransomware gang sites over TOR, doxxing and hate platforms in form search API's and scraped data that be can used to quickly form correlation between different threat actor movements and discussions.\n\nA comprehensive cybersecurity intelligence platform that aggregates, analyzes, and provides searchable access to data from various sources including dark web forums, breach databases, and ransomware posts.\n\n**Note The documentation of this project is still under work and will be updated accordingly**\n\n## Overview\n\nTAP is designed to help security professionals monitor and analyze cyber threats by collecting and indexing data from multiple sources:\n\n- Breach database leaks \n- Doxbin records\n- Nulled forum data\n- OnniForums intelligence\n- Cracked accounts and combos\n- Ransomware group activities\n- IOC (Indicators of Compromise) checking\n- IP reputation analysis\n- File hash verification\n\nThe platform provides both a web interface for interactive searches and API endpoints for integration with other security tools.\n\n## Architecture\n\nThe system consists of multiple components:\n\n- **Frontend**: React-based UI with Tailwind CSS and Material UI\n- **Admin Panel**: Management interface for system administration\n- **APIs**: Python and Go services for data retrieval and processing\n- **Container**: Dockerized environment for dark web scraping\n- **Database**: PostgreSQL database for storing indexed data\n- **Data Loaders**: Scripts to collect and process data\n- **NGINX**: Reverse proxy for routing traffic to different services\n- **Docker**: Containerization for easy deployment and scaling\n\n### System Architecture\n\n![System Design](./images/System%20Design.png)\n\n## Dataset\nYou can download the data that has already been scraped by the platform at [kaggle.com/snowkluster](https://www.kaggle.com/datasets/snowkluster/dark-web-posts/data)\n\n## Getting Started\n\n### Prerequisites\n\n- Docker and Docker Compose\n- Node.js 22+\n- Python 3.11+\n- Go 1.21+\n\n### Installation\n\n1. Clone the repository:\n    ```bash\n    git clone https://github.com/snowkluster/TAP.git\n    cd TAP\n    ```\n\n2. Start dashboard frontend:\n    ```bash\n    cd frontend\n    npm install\n    npm run dev\n    ```\n\n3. Start admin panel [frontend]:\n    ```bash\n    cd admin/frontend\n    npm install\n    npm run dev\n    ```\n\n4. Start admin panel [backend]:\n    ```bash\n    cd admin/backend\n    npm install\n    node app.js\n    ```\n\n5. Start API:\n    ```bash\n    pip install -r requirements.txt\n    cd api\n    ./build.sh\n    ./starter.sh\n    ```\n\n6. Start the platform using Docker Compose:\n   ```bash\n   ./start.sh\n   ```\n\n7. Access the services:\n   - Main Portal: http://portal.localhost\n   - Admin Dashboard: http://dashboard.localhost\n\n### Docker Configuration\n\nThe project uses Docker Compose to orchestrate multiple services:\n\n- **NGINX**: Reverse proxy for routing traffic\n- **PostgreSQL**: Database for storing collected data\n- **SQLite**: Database for backup storage incase of failure of PSQL Database\n\nThe `compose.yml` file defines these services and their configurations.\n\n### Networking\n\nThe platform uses two networking approaches:\n- **Host Network**: For the NGINX service, allowing direct access to localhost services\n- **Bridge Network**: For the database and other services, providing container isolation\n\n### Development Setup\n\nFor frontend development:\n```bash\ncd frontend\nnpm install\nnpm run dev\n```\n\nFor admin panel development [frontend]:\n```bash\ncd admin/frontend\nnpm install\nnpm run dev\n```\n\nFor admin panel development [backend]:\n```bash\ncd admin/backend\nnpm install\nnode app.js\n```\n\nFor API development:\n```bash\ncd api\n./build.sh\n./starter.sh\n```\n\n## Features\n\n- **Live Search**: Query across multiple data sources\n- **Breach Search**: Find compromised credentials\n- **Darknet Feed**: Monitor latest dark web activities\n- **Ransomware Post Tracking**: Stay updated on ransomware group activities\n- **IP \u0026 Hash Analysis**: Check reputation and malware indicators\n- **Cybersecurity News**: Latest updates from the security community\n\n## API Documentation\n\nThe platform provides several API endpoints:\n\n- `:8010/search/\"`: Search breached forums for records\n- `:8002/search/`: Search doxbin forums for records\n- `:8013/search/`: Search nulled forums for records\n- `:8014/search/`: Search OnniForums forums for records\n- `8004:/check_ip/?ip=:\u003cIP_ADDR\u003e`: Check IP reputation\n- `8006:/check/:\u003cFILE_HASH\u003e`: Verify file hashes and reputation\n- `8009:/scrape/`: Get latest ransomware posts\n- `:8008/`: Check indicators of compromise\n\nFor further details look at [API Docs](./docs/API.md)\n\n## Server Configuration\n\n### NGINX\n\nThe platform uses NGINX as a reverse proxy to route traffic to different services:\n\n- `portal.localhost` routes to the main frontend application\n- `dashboard.localhost` routes to the admin dashboard\n\nThe NGINX configuration is stored in `public/nginx.conf` and is mounted into the NGINX container.\n\n### Database\n\nPostgreSQL is used as the primary database with the following configuration:\n\n- **Username**: dbuser\n- **Database**: darkweb\n- **Port**: 5432\n\nData persistence is managed via Docker volumes.\n\n## Maintenance\n\n### Cleaning Scripts\n\nThe repository includes several utility scripts:\n- `clean.sh`: General cleanup\n- `clean-api.sh`: API-specific cleanup\n- `wlc.sh`: Welcome script\n\n## Security\n\nThis platform is designed for legitimate security research and threat intelligence purposes only. See [SECURITY.md](SECURITY.md) for security policies and responsible usage guidelines.\n\n## Contributing\n\nContributions are welcome! Please check the [TODO.md](TODO.md) file for areas that need help. Follow these steps:\n\n1. Fork the repository\n2. Create a feature branch (`git checkout -b feature/amazing-feature`)\n3. Commit your changes (`git commit -m 'Add some amazing feature'`)\n4. Push to the branch (`git push origin feature/amazing-feature`)\n5. Open a Pull Request\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n\n## Authors\n\n- [@snowkluster](https://www.github.com/snowkluster)\n- [@aparna2573](https://github.com/aparna2573)\n- [@geetansh14](https://github.com/geetansh14)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnowkluster%2Ftap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsnowkluster%2Ftap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnowkluster%2Ftap/lists"}