{"id":18360120,"url":"https://github.com/cristiangreco/gcrawler","last_synced_at":"2025-07-30T16:34:07.406Z","repository":{"id":138551436,"uuid":"90093941","full_name":"cristiangreco/gcrawler","owner":"cristiangreco","description":"A simple (not concurrent) web crawler written in Java.","archived":false,"fork":false,"pushed_at":"2021-01-15T14:55:20.000Z","size":70,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-10T04:18:17.370Z","etag":null,"topics":["crawler","java"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cristiangreco.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-05-03T01:14:58.000Z","updated_at":"2025-01-13T08:22:37.000Z","dependencies_parsed_at":"2023-05-24T17:30:27.502Z","dependency_job_id":null,"html_url":"https://github.com/cristiangreco/gcrawler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cristiangreco/gcrawler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cristiangreco%2Fgcrawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cristiangreco%2Fgcrawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cristiangreco%2Fgcrawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cristiangreco%2Fgcrawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cristiangreco","download_url":"https://codeload.github.com/cristiangreco/gcrawler/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cristiangreco%2Fgcrawler/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267899153,"owners_count":24162991,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-30T02:00:09.044Z","response_time":70,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","java"],"created_at":"2024-11-05T22:26:40.130Z","updated_at":"2025-07-30T16:34:07.365Z","avatar_url":"https://github.com/cristiangreco.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"gcrawler\n===\n\n[![Build Status](https://travis-ci.org/cristiangreco/gcrawler.svg?branch=master)](https://travis-ci.org/cristiangreco/gcrawler)\n\n\u003e :warning: :fire: A pull request will be merged soon that adds [concurrency support](https://github.com/cristiangreco/gcrawler/pull/1). :tada:\n\ngcrawler is a simple (not concurrent) web crawler written in Java.\n\nGiven a starting url, it visits every reachable page under that domain, not crossing subdomains.\n\ngcrawler outputs a json to stdout listing every visited page and every static asset (images, js, css) found on that page.\n\nExample output:\n\n```\n[\n  {\n    \"url\": \"http://www.debian.org\",\n    \"assets\": [\n      \"http://www.debian.org/Pics/openlogo-50.png\",\n      \"http://www.debian.org/Pics/identica.png\",\n      \"http://www.debian.org/Pics/planet.png\",\n      \"http://www.debian.org/debhome.css\",\n      \"http://www.debian.org/debian-en.css\",\n      \"http://www.debian.org/favicon.ico\"\n    ]\n  },\n  {\n    \"url\": \"http://www.debian.org/\",\n    \"assets\": [\n      \"http://www.debian.org/Pics/openlogo-50.png\",\n      \"http://www.debian.org/Pics/identica.png\",\n      \"http://www.debian.org/Pics/planet.png\",\n      \"http://www.debian.org/debhome.css\",\n      \"http://www.debian.org/debian-en.css\",\n      \"http://www.debian.org/favicon.ico\"\n    ]\n  },\n  {\n    \"url\": \"http://www.debian.org#content\",\n    \"assets\": [\n      \"http://www.debian.org/Pics/openlogo-50.png\",\n      \"http://www.debian.org/Pics/identica.png\",\n      \"http://www.debian.org/Pics/planet.png\",\n      \"http://www.debian.org/debhome.css\",\n      \"http://www.debian.org/debian-en.css\",\n      \"http://www.debian.org/favicon.ico\"\n    ]\n  },\n  {\n    \"url\": \"http://www.debian.org/intro/about\",\n    \"assets\": [\n      \"http://www.debian.org/Pics/openlogo-50.png\",\n      \"http://www.debian.org/Pics/identica.png\",\n      \"http://www.debian.org/Pics/planet.png\",\n      \"http://www.debian.org/debian.css\",\n      \"http://www.debian.org/debian-en.css\"\n    ]\n  },\n  ...\n]\n```\n\n## How to build\n\ngcrawler uses Gradle as build system. You don't need to install Gradle as it is shipped with sources.\n\nThe following will create an executable uber-jar under `build/libs/gcrawler-all.jar`:\n\n```sh\n$ ./gradlew clean shadowJar\n```\n\n## How to run\n\nRun the gcrawler executable jar passing the starting url as parameter:\n\n```sh\n$ java -jar build/libs/gcrawler-all.jar http://www.debian.org\n```\n\ngcrawler supports some command line options, see below.\n\n## High level design\n\ngcrawler is written in Java and uses the fantastic [JSoup](https://jsoup.org) library to fetch and parse HTML documents, plus the [Jackson](https://github.com/FasterXML/jackson) json serializer.\n\nThe implementation is single-threaded by purpose, but it could easily be parallelized with minor changes.\n\nCPU usage is very low because the main cpu-intensive activity (DOM parsing) happens only in the main thread.\n\nMemory usage is generally low, because documents are not retained after parsing and json results are emitted as soon as possible. In fact, gcrawler uses the Jackson streaming APIs to flush output as soon as documents are parsed.\n\n## Stopping the crawler\n\ngcrawler will try hard to emit a valid json to stdout.\n\nUpon task cancellation (e.g. when hitting ^C) the crawler will stop as soon as possible (i.e. at the end of the currently running task) and will emit a valid json for the pages successfully visited.\n\n## Error handling\n\nAgain, gcrawler will try hard to emit a valid json to stdout and, in general, it will avoid to intermingle stacktraces with json output.\n\nErrors are printed to stderr instead.\n \nErrors happening before any json character has been emitted (e.g. wrong command line parameters), will be printed to stderr and will cause the application to exit early.\n\nErrors happening while the crawler is running and json is being printed, are muted by default. To enable printing crawling errors to stderr use the `--print-errors true` flag. \n\n## Command line parameters\n\ngcrawler requires an odd number of command line parameters, e.g.:\n\n```sh\n$ java -jar build/libs/gcrawler-all.jar [param1 value1 param2 value2 ...] \u003curl\u003e\n```\n\nEach parameter must be followed by its value. The starting url must be passed as last parameter.\n\nflag | values | default | description\n---- | ------ | ------- | -----------\n`--print-errors` | boolean | false | Print any exception to stderr\n`--halt-on-error` | boolean | false | Stop crawling after any error occurs\n`--normalize-urls` | boolean | true | Normalize urls before crawling (remove traling `#` and `?`) \n`--timeout-millis` | integer | 5000 | Socket timeout in milliseconds for connect and read operations\n\nFor example, the following:\n\n```sh\njava -jar build/libs/gcrawler-all.jar --print-errors true --timeout-millis 10000 www.debian.org 2\u003eerrors.txt \u003eoutput.txt\n```\n\nwill apply a connection timeout of 10 seconds, will print to file `errors.txt` any skipped url and will save the json to `output.txt`. \n\n## Notes\n\n- In case the starting url returns any error, the empty json list `[]` is printed.\n- In case a page does not import any static asset, the empty json list `\"assets\":[]` is printed.\n- gcrawler does not apply any url normalization, except for removing trailing fragment `#` and question mark `?` in order to avoid too much duplicate url. This can be configured with the `--normalize-urls` parameter.\n- At the moment, it is possible to configure only the JSoup connection timeout. Anyway JSoup provides sensible defaults: it will follow redirects, will not ignore http errors, will validate certificates, and will handle response body sizes up to 1M.\n- gcrawler will cross between http and https without problems, but will refuse to cross between domains, therefore crawling `debian.org` is different than crawling `www.debian.org`. \n\n## License\n\nThe gcrawler source files are distributed under the BSD-style license.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcristiangreco%2Fgcrawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcristiangreco%2Fgcrawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcristiangreco%2Fgcrawler/lists"}