{"id":19637187,"url":"https://github.com/kenych/java-web-crawler","last_synced_at":"2025-04-28T09:31:30.831Z","repository":{"id":198655825,"uuid":"85333155","full_name":"kenych/java-web-crawler","owner":"kenych","description":"This is pretty basic example of web page crawling in java and is not fully production ready crawler and is done for test purposes only","archived":false,"fork":false,"pushed_at":"2022-09-01T22:31:56.000Z","size":12,"stargazers_count":3,"open_issues_count":2,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-05T08:04:51.527Z","etag":null,"topics":["concurrency","java8","web-crawler"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kenych.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2017-03-17T16:33:33.000Z","updated_at":"2024-09-11T18:16:57.000Z","dependencies_parsed_at":null,"dependency_job_id":"b5c4db9d-6f6b-4788-9c00-05fc4273b7fe","html_url":"https://github.com/kenych/java-web-crawler","commit_stats":null,"previous_names":["kenych/java-web-crawler"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kenych%2Fjava-web-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kenych%2Fjava-web-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kenych%2Fjava-web-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kenych%2Fjava-web-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kenych","download_url":"https://codeload.github.com/kenych/java-web-crawler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251284882,"owners_count":21564688,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurrency","java8","web-crawler"],"created_at":"2024-11-11T12:33:33.853Z","updated_at":"2025-04-28T09:31:25.818Z","avatar_url":"https://github.com/kenych.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"This is pretty basic example of web page crawling in java and is not fully production ready crawler and is done for test purposes only.\n\nRunning code against different web sites showed\nthere are some issues which could be improved.\nFor example it can't follow relative links at current stage, also some web sites\n have dynamically generated links, which actually might refer to same page, so if this happens\n then application will run in a loop and will never finish.\nThere could be other issues as well, in fact as it is crawling external resources which could be very various,\nit is hard to cover all cases.\n\nOther issues could be regarding retry mechanism which is missing, better error handling, etc etc.\nBut for the sake of this test I have skipped them.\n\nCode is covered by tests and example web server with two pages(crawl.html, crawl2.html)\n\nThe code is written in concurrent way and using given number of threads.\n\nTo test the application either run CrawlerTest tests or commands below:\nMaven\nmvn clean package \u0026\u0026 java -jar target/crawler-1.0-SNAPSHOT.jar https://github.com/kenych https://github.com/kenych 5\n\nDocker\nBuild image locally with maven and docker:\nmvn clean package \u0026\u0026 docker build  -t web-crawler:local . \u0026\u0026 docker run --rm  web-crawler:local  https://github.com/kenych https://github.com/kenych 5\n\nor with Maven plugin:\nmvn clean package docker:build \u0026\u0026 docker run --rm  web-crawler:local  https://github.com/kenych https://github.com/kenych 5\n\nOr get it from dockerHub:\ndocker run --rm  kayan/web-crawler  https://github.com/kenych https://github.com/kenych 5\n\n\nWith regards,\nKayan A.\n\n\n\n\n\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkenych%2Fjava-web-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkenych%2Fjava-web-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkenych%2Fjava-web-crawler/lists"}