{"id":28626245,"url":"https://github.com/commoncrawl/cc-nutch-example","last_synced_at":"2025-06-12T08:41:15.545Z","repository":{"id":53674045,"uuid":"279346635","full_name":"commoncrawl/cc-nutch-example","owner":"commoncrawl","description":"Apache Nutch example project to archive content in WARC files","archived":false,"fork":false,"pushed_at":"2022-12-22T12:32:33.000Z","size":8,"stargazers_count":3,"open_issues_count":1,"forks_count":2,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-05-07T18:23:39.076Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/commoncrawl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-07-13T15:50:46.000Z","updated_at":"2023-08-12T14:51:57.000Z","dependencies_parsed_at":"2023-01-30T08:46:21.602Z","dependency_job_id":null,"html_url":"https://github.com/commoncrawl/cc-nutch-example","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/commoncrawl/cc-nutch-example","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/commoncrawl%2Fcc-nutch-example","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/commoncrawl%2Fcc-nutch-example/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/commoncrawl%2Fcc-nutch-example/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/commoncrawl%2Fcc-nutch-example/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/commoncrawl","download_url":"https://codeload.github.com/commoncrawl/cc-nutch-example/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/commoncrawl%2Fcc-nutch-example/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259432365,"owners_count":22856734,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-12T08:41:14.233Z","updated_at":"2025-06-12T08:41:15.536Z","avatar_url":"https://github.com/commoncrawl.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"Example Usage of Common Crawl's Fork of Apache Nutch to Crawl and Write WARC files\n==================================================================================\n\nA short description how to set up [Common Crawl's\nFork](https://github.com/commoncrawl/nutch/) of [Apache\nNutch](https://nutch.apache.org/) for crawling and to store the\ncrawled content in WARC files.\n\n\n# Requirements and installation\n\n- Linux (tested on Ubuntu 20.04)\n- Java 11 (higher Java versions should also work)\n- [ant](https://ant.apache.org/) and [maven](https://maven.apache.org/)\n- [Compact Language Detector 2](https://github.com/CLD2Owners/cld2)\n\n\n```bash\nsudo apt install libcld2-0 libcld2-dev ant maven\n```\n\n# Compile Nutch and required projects\n\n```bash\ngit clone git@github.com:crawler-commons/crawler-commons.git\ncd crawler-commons/\nmvn install\ncd ..\n\ngit clone git@github.com:commoncrawl/language-detection-cld2.git\ncd language-detection-cld2/\nmvn install\ncd ..\n\ngit clone https://github.com/commoncrawl/nutch.git nutch-cc\ncd nutch-cc/\nant runtime\ncd ..\n```\n\n# Configuration\n\nGo to the project root folder `nutch-cc` and edit the files in the\nfolder `conf/` esp. `conf/nutch-site.xml`.  But also the URL filter\nconfiguration files may require to be adapted to your use case.\n\nNotes:\n- it's required to configure at least the property `http.agent.name`\n  in the file `conf/nutch-site.xml`\n- if the configuration is changed Nutch needs to be recompiled because\n  configuration files are contained in the job file (`runtime/local/apache-nutch-*.job`)\n\n\n\n# Run crawl\n\n```bash\necho -e \"https://nutch.apache.org/\\tnutch.score=1.0\" \u003eurls.txt\n\n./crawl.sh crawl 3 urls.txt\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcommoncrawl%2Fcc-nutch-example","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcommoncrawl%2Fcc-nutch-example","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcommoncrawl%2Fcc-nutch-example/lists"}