{"id":35094644,"url":"https://github.com/cryxnet/crawnet","last_synced_at":"2026-04-10T13:31:35.799Z","repository":{"id":122724679,"uuid":"558400557","full_name":"cryxnet/crawnet","owner":"cryxnet","description":"CRAWNET is a graph-based domain discovery tool by CRYXNET that helps you gather information about domains and potential relationships with other actors.","archived":false,"fork":false,"pushed_at":"2023-07-25T21:36:38.000Z","size":758,"stargazers_count":4,"open_issues_count":4,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-12-29T07:25:12.930Z","etag":null,"topics":["cybersecurity","data-mining","docker","flask","fullstack","graph","information-gathering","neo4j","nextjs","osint","python","red-teaming","security"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cryxnet.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-10-27T13:28:19.000Z","updated_at":"2025-09-06T22:09:38.000Z","dependencies_parsed_at":null,"dependency_job_id":"047ad735-a165-4324-af7a-10c713d32484","html_url":"https://github.com/cryxnet/crawnet","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/cryxnet/crawnet","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cryxnet%2Fcrawnet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cryxnet%2Fcrawnet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cryxnet%2Fcrawnet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cryxnet%2Fcrawnet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cryxnet","download_url":"https://codeload.github.com/cryxnet/crawnet/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cryxnet%2Fcrawnet/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31645214,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-10T07:40:12.752Z","status":"ssl_error","status_checked_at":"2026-04-10T07:40:11.664Z","response_time":98,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cybersecurity","data-mining","docker","flask","fullstack","graph","information-gathering","neo4j","nextjs","osint","python","red-teaming","security"],"created_at":"2025-12-27T15:07:52.087Z","updated_at":"2026-04-10T13:31:35.712Z","avatar_url":"https://github.com/cryxnet.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cbr /\u003e\n\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://github.com/cryxnet/crawnet\"\u003e\n    \u003cimg src=\"assets/crawnet.png\" alt=\"Logo\" width=\"300\" height=\"120\"\u003e\n  \u003c/a\u003e\n\n  \u003ch3 align=\"center\"\u003eCRAWNET\u003c/h3\u003e\n\n  \u003cp align=\"center\"\u003e\nCRAWNET is a graph-based domain discovery tool that helps you gather information\nabout domains and potential relationships with other actors.\nWith the power of graph databases, we can find and display quickly information and relationship with other actors, to identify threats before they happen and much more. This tool is perfect for digital forensic, red teaming, penetration testing and cybersecurity researchers.\n    \u003cbr /\u003e\n    \u003ca href=\"https://github.com/cryxnet/crawnet\"\u003e\u003cstrong\u003eExplore the docs »\u003c/strong\u003e\u003c/a\u003e\n    \u003cbr /\u003e\n    \u003cbr /\u003e\n    \u003ca href=\"https://github.com/cryxnet/crawnet/issues\"\u003eReport Bug\u003c/a\u003e\n    ·\n    \u003ca href=\"https://github.com/cryxnet/crawnet/issues\"\u003eRequest Feature\u003c/a\u003e\n        ·\n    \u003ca href=\"https://github.com/cryxnet/crawnet/security/policy\"\u003eReport Vulnerability\u003c/a\u003e\n  \u003c/p\u003e\n\u003c/div\u003e\n\n## Table of Contents\n\n-   [Table of Contents](#table-of-contents)\n-   [Installation](#installation)\n-   [Roadmap](#roadmap)\n-   [Intelligence Data](#intelligence-data)\n    -   [Subdomains](#subdomains)\n-   [Architecture](#architecture)\n-   [Testing](#testing)\n    -   [Local Testing](#local-testing)\n-   [Pipelines](#pipelines)\n-   [Security Policy](#security-policy)\n-   [Hosting in the Cloud](#hosting-in-the-cloud)\n-   [Disclaimer](#disclaimer)\n-   [Author](#author)\n\n## Installation\n\nTo install the project and its dependencies, follow these steps:\n\n**Make sure you have docker installed!**\n\n1. Clone the repository to your local machine:\n\n```bash\ngit clone https://github.com/cryxnet/crawnet.git\n```\n\n2. Go to the directory\n\n```bash\ncd crawnet\n```\n\n3. Rename (to `.env`) and edit the [configuration file](/sample.env)\n\n```bash\nmv sample.env .env\n\u0026\u0026 sed -i 's/NEO4J_PASSWORD=CHANGEME/NEO4J_PASSWORD=your_password_here/' .env \\\n\u0026\u0026 sed -i 's/FLASK_APP_URL=http:\\/\\/CHANGEME:5000/FLASK_APP_URL=http:\\/\\/your_machine_ip_or_localhost:5000/' .env \\\n\u0026\u0026 sed -i 's/FLASK_DEBUG=1/FLASK_DEBUG=0/' .env\n```\n\n4. Start the docker stack\n\n```bash\ndocker compose -f docker-compose.prod.yaml up\n```\n\n## Roadmap\n\n**v1**\n\n-   [x] The overseen service (the api) can collect data from various open source intelligence services.\n-   [x] The overseen service (the api) can create entity nodes and relationships in neo4j.\n-   [x] The user can easily interact with the api\n-   [x] The user can submit a new domain to discover via an formular\n-   [x] The user has an interactive way to see the nodes and their relationships on the dashboard\n-   [x] First Version Release (09.05.2023)\n\n**v2**\n(Development begins Q3 2023)\n\n-   [ ] Collectors Handling (Easy to add new intelligence and entity nodes)\n-   [ ] Relationship detection \u0026 connection engine (Database engine that automatically connects nodes if they have anything in common together)\n-   [ ] Threat Intelligence Data\n-   [ ] Better UI/UX\n\n## Intelligence Data\n\nThe Intelligence Service is using the following sources:\n\n-   [Cert Fingerprint | CRT.SH](https://crt.sh/)\n-   [IP Information | IPAPI.CO](https://ipapi.co/IPADDRESS/json/)\n-   [Python | WHOIS](https://pypi.org/project/python-whois/)\n\n### Subdomains\n\nTo gather subdomains from a domain we use technique like google dorks and certifcate fingerprinting.\n\n## Architecture\n\n\u003cimg src=\"assets/crawnet_microservice_architecture.png\" alt=\"architecture\" heigth=\"400\" width=\"400\"\u003e\n\n-   For the dashboard we use [Next.js](https://nextjs.org/) with the [MUI](https://mui.com/), [vis-data](https://www.npmjs.com/package/vis-data) and [vis-network](https://www.npmjs.com/package/vis-network) libraries (we also use other libraries). vis-data and vis-network is needed to display the nodes and their relationships.\n-   For the API we use the Flask framework for python.\n-   For the database we use the graph database [Neo4j](https://neo4j.com/)\n\n## Testing\n\n-   We have a pipeline that is automatically testing the [intergration](/tests/integration.py) between the 3 services and source code for any vulnerabilities\n\n| Test Case                 | Expected Outcome                                                                        | Outcome                                                                                                                                                             |\n| ------------------------- | --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| Reach Website             | Response status code is 200 and \"Success\" is printed to console                         | [![Integration Test](https://github.com/cryxnet/crawnet/actions/workflows/testing.yml/badge.svg)](https://github.com/cryxnet/crawnet/actions/workflows/testing.yml) |\n| Reach API                 | Response status code is 200 or 404 and \"Success\" is printed to console                  | [![Integration Test](https://github.com/cryxnet/crawnet/actions/workflows/testing.yml/badge.svg)](https://github.com/cryxnet/crawnet/actions/workflows/testing.yml) |\n| Reach Neo4j               | Neo4j connection is successful and \"Success\" is printed to console                      | [![Integration Test](https://github.com/cryxnet/crawnet/actions/workflows/testing.yml/badge.svg)](https://github.com/cryxnet/crawnet/actions/workflows/testing.yml) |\n| API-Website Communication | Response status code is 200 and \"Success\" is printed to console                         | [![Integration Test](https://github.com/cryxnet/crawnet/actions/workflows/testing.yml/badge.svg)](https://github.com/cryxnet/crawnet/actions/workflows/testing.yml) |\n| API-Neo4j Communication   | Response JSON is {'nodes': [], 'relationships': []} and \"Success\" is printed to console | [![Integration Test](https://github.com/cryxnet/crawnet/actions/workflows/testing.yml/badge.svg)](https://github.com/cryxnet/crawnet/actions/workflows/testing.yml) |\n| Vulnerable Python Code    | Nothing Found                                                                           | ![Actions Status](https://github.com/cryxnet/crawnet/workflows/CodeQL/badge.svg)                                                                                    |\n| Vulnerable TS Code        | Nothing Found                                                                           | ![Actions Status](https://github.com/cryxnet/crawnet/workflows/CodeQL/badge.svg)                                                                                    |\n\n### Local Testing\n\n```bash\npython tests/intergration.py\n```\n\n## Pipelines\n\n-   [Testing Pipeline](https://github.com/cryxnet/crawnet/actions/workflows/testing.yml): This pipeline tests our tool with integration tests.\n-   [CodeQL Pipeline](https://github.com/cryxnet/crawnet/actions/workflows/github-code-scanning/codeql): This pipeline tests for any vulnerability in our code\n-   [Docker Image Push Pipeline](https://github.com/cryxnet/crawnet/actions/workflows/docker-image.yml): This pipelines pushes a new docker image to the dockerhub registry when there is a new release.\n\n## Security Policy\n\n-   Please take a moment to review and familiarize yourself with our [security policy](/SECURITY.md).\n-   We encourage you to report any issues, bugs, or vulnerabilities that you encounter while using our service.\n-   Your help in identifying and reporting any security issues or vulnerabilities is greatly appreciated. Please refer to our [security policy](/SECURITY.md) for guidance.\n\n## Hosting in the Cloud\n\nA short guide if you wan't to deploy this tool for your team in the cloud. This are the minimum requirements.\n\n| Cloud Provider | Instance Type | vCPUs | RAM | Storage | Cost (ca. per Hour) |\n| -------------- | ------------- | ----- | --- | ------- | ------------------- |\n| AWS            | t4g.medium    | 2     | 4GB | 20GB    | $0.0336             |\n| Azure          | A2 v2         | 2     | 4GB | 20GB    | $0.136              |\n\n## Disclaimer\n\nYOUR USAGE OF THIS PROJECT CONSTITUTES YOUR AGREEMENT TO THE FOLLOWING TERMS:\n\n    THE MISUSE OF THE DATA PROVIDED BY THIS PROJECT MAY LEAD TO CRIMINAL CHARGES AGAINST THE PERSONS CONCERNED.\n\n    I DO NOT TAKE ANY RESPONSIBILITY FOR THE CASE. USE THIS PROJECT ONLY FOR RESEARCH PURPOSES, EDUCATIONAL PURPOSES \u0026 ETHICAL ONLY.\n\n    CRAWNET is a project related to Computer Security and for Educational Purposes and not a project that promotes illegal activities.\n\n    Don't use this Project for any illegal activities.\n\n    If something happens, we do not take any liability.\n\n    CRAWNET should be considered as a project for educational purposes.\n\n## Author\n\nCreated by [cryxnet](https://cryxnet.com/)\n\nIf you find this project helpful, please give it a ⭐️ on GitHub to show your support.\nI would also appreciate it if you shared it with others who might find it useful!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcryxnet%2Fcrawnet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcryxnet%2Fcrawnet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcryxnet%2Fcrawnet/lists"}