{"id":30288340,"url":"https://github.com/hafidsousa/webcrawler","last_synced_at":"2025-08-16T22:04:15.592Z","repository":{"id":268510598,"uuid":"103866443","full_name":"hafidsousa/webcrawler","owner":"hafidsousa","description":"A Recursive Web crawler built with Java 8, reactive streams, async queues and AWS DynamoDB.","archived":false,"fork":false,"pushed_at":"2017-09-29T06:08:49.000Z","size":555,"stargazers_count":7,"open_issues_count":0,"forks_count":4,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-12-17T08:22:49.740Z","etag":null,"topics":["aws-dynamodb","aws-sdk","aws-sqs","java","reactive-programming","reactive-streams"],"latest_commit_sha":null,"homepage":"https://github.com/hafidsousa/webcrawler","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hafidsousa.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-09-17T22:28:09.000Z","updated_at":"2023-08-13T12:12:13.000Z","dependencies_parsed_at":"2024-12-17T08:34:09.717Z","dependency_job_id":null,"html_url":"https://github.com/hafidsousa/webcrawler","commit_stats":null,"previous_names":["hafidsousa/webcrawler"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/hafidsousa/webcrawler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hafidsousa%2Fwebcrawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hafidsousa%2Fwebcrawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hafidsousa%2Fwebcrawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hafidsousa%2Fwebcrawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hafidsousa","download_url":"https://codeload.github.com/hafidsousa/webcrawler/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hafidsousa%2Fwebcrawler/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270775881,"owners_count":24642964,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-16T02:00:11.002Z","response_time":91,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-dynamodb","aws-sdk","aws-sqs","java","reactive-programming","reactive-streams"],"created_at":"2025-08-16T22:03:37.333Z","updated_at":"2025-08-16T22:04:15.566Z","avatar_url":"https://github.com/hafidsousa.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# A Reactive Web Crawler\n\nA Recursive Web crawler built with Java 8, reactive streams, async queues and AWS DynamoDB.\n\n## Problem\nCrawling Web sites is an intensive task, and it comes with some challenges.\n\n* How to crawl sub-pages when the website hierarchy is unknown?\n* How to crawl websites efficiently?\n* How to manage long-running HTTP requests?\n\n## Diagram v1\n![Diagram](public/images/crawler_v1.png)\n\n## Implementation\n* A version the [Depth-first search (DFS)](https://en.wikipedia.org/wiki/Depth-first_search) algorithm has been implemented. The search starts from the seed URL and traverses the sub-pages until the pre-defined depth limit is reached, or there are no more sub-pages.\n* Using [netty.io](https://netty.io/) client with non-blocking calls and multiple threads.\n* If the Seed URL exists:\n  * Reply with existing record.\n* If the Seed URL doesn't exist:\n  * Create a record in a NEW state and reply to the API Consumer.\n  * Send the request to the queue.\n  * The batch server picks up the request and updates the record to IN_PROGRESS.\n  * Once the task is completed save the record to the database. If successful set status to COMPLETED, otherwise FAILED.\n\n### Limitations\n- [TODO] Read and Respect robots.txt [Version 2](#diagram-v2-todo).\n- [TODO] The API Consumer is not notified when the crawling process is completed [Version 2](#diagram-v2-todo).\n- [TODO] In-Memory cache to reduce response time. [Version 2](#diagram-v2-todo).\n- [TODO] API Security mechanisms: Request logging, throttling and auditing [Version 2](#diagram-v2-todo).\n- [TODO] Automated CI/CD Pipelines [Version 2](#diagram-v2-todo).\n- [TODO] UI [Version 2](#diagram-v2-todo).\n- [TODO] Add contract, component, integration and performance tests [Version 2](#diagram-v2-todo).\n- Crawling depth must be defined.\n- Only nodes prefixed with the Seed URL will be recursed.\n\u003e **Example:**\n\u003e\n\u003e seedUrl = http://first-example.com\n\u003e\n\u003e nodes = \"http://first-example.com/about\", \"http://another-example.com\"\n\u003e\n\u003e Only URLs prefixed with 'http://first-example.com' will be crawled recursively.\n\n## Getting Started\n\nThese instructions will get you a copy of the project up and running on your **local machine** for development and testing purposes.\n\n### Prerequisites\n\n```\nJava8\nGradle 4.1 (Or just use provided Gradle wrapper)\nTo Run the application in a Integrated Environment you must configure AWS DynamoDB and SQS endpoints at `src/main/resources/application.properties`.\n\n```\n## Build\n\n```\ngradle clean build\n```\n\n## Running Unit tests\n\n```\ngradle test\n```\n\n## Built With\n\n* [Gradle 4.1](https://docs.gradle.org/4.1/userguide/userguide.html) - Dependency Management\n* [Spring Boot 2](https://docs.spring.io/spring-boot/docs/2.0.0.M4/reference/htmlsingle/) - Web framework\n\t* [Spring 5 Web Reactive](https://docs.spring.io/spring-framework/docs/5.0.0.M4/spring-framework-reference/htmlsingle/) - Spring Support for Web Reactive\n\t* [Reactor](https://projectreactor.io/) - Reactive Core\n\t* [Netty io]( https://netty.io/) - Web Client\n\t* [Spring JMS](https://docs.spring.io/spring/docs/current/spring-framework-reference/html/jms.html) - Java Messaging Service\n* [AWS SDK](https://aws.amazon.com/sdk-for-java/) - AWS SDK for Java\n* [AWS SQS](https://aws.amazon.com/sqs/) - Messaging Provider\n* [AWS DynamoDB](https://aws.amazon.com/documentation/dynamodb/) - NoSQL Database\n* [Jsoup](https://jsoup.org/) - Java HTML Parser\n* [Wiremocks](http://wiremock.org/) - Mock URLs for testing\n\n## Diagram v2 (TODO)\n![Diagram](public/images/crawler_v2.png)\n\n* Add Redis In-Memory cache.\n* Add AWS API Gateway (API Management).\n* Add Lambda function to be triggered on DynamoDB events.\n* Add SNS Notification.\n* Infrastructure as a code using AWS Cloud Formation.\n* Pipeline as a Code using Jenkis.\n* Angular 4 UI.\n\n## Author\n\n* **[Hafid F Sousa](https://github.com/hafidsousa)**\n\n## License\n\nThis project is licensed under the [Apache License 2.0](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhafidsousa%2Fwebcrawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhafidsousa%2Fwebcrawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhafidsousa%2Fwebcrawler/lists"}