{"id":19316578,"url":"https://github.com/tokenmill/crawling-framework","last_synced_at":"2025-04-22T17:30:24.784Z","repository":{"id":25183644,"uuid":"103384901","full_name":"tokenmill/crawling-framework","owner":"tokenmill","description":"Easily crawl news portals or blog sites using Storm Crawler.","archived":false,"fork":false,"pushed_at":"2022-11-15T23:31:06.000Z","size":940,"stargazers_count":20,"open_issues_count":21,"forks_count":4,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-02T02:02:02.188Z","etag":null,"topics":["crawler","crawling","crawling-framework","elasticsearch","java","scraping","storm","storm-crawler","vaadin"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tokenmill.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-09-13T10:12:37.000Z","updated_at":"2024-09-07T13:52:17.000Z","dependencies_parsed_at":"2023-01-14T02:16:44.903Z","dependency_job_id":null,"html_url":"https://github.com/tokenmill/crawling-framework","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokenmill%2Fcrawling-framework","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokenmill%2Fcrawling-framework/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokenmill%2Fcrawling-framework/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tokenmill%2Fcrawling-framework/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tokenmill","download_url":"https://codeload.github.com/tokenmill/crawling-framework/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250287337,"owners_count":21405588,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","crawling","crawling-framework","elasticsearch","java","scraping","storm","storm-crawler","vaadin"],"created_at":"2024-11-10T01:11:56.455Z","updated_at":"2025-04-22T17:30:24.123Z","avatar_url":"https://github.com/tokenmill.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ca href=\"http://www.tokenmill.lt\"\u003e\n      \u003cimg src=\".github/tokenmill-logo.svg\" width=\"125\" height=\"125\" align=\"right\" /\u003e\n\u003c/a\u003e\n\n# Crawling Framework\n\n[![Maven Central](https://img.shields.io/maven-central/v/lt.tokenmill.crawling/crawling-framework.svg?label=Maven%20Central)](https://search.maven.org/search?q=g:%22lt.tokenmill.crawling%22%20AND%20a:%22crawling-framework%22)\n[![pipeline status](https://gitlab.com/tokenmill/crawling-framework/badges/master/pipeline.svg)](https://gitlab.com/tokenmill/crawling-framework/commits/master)\n\nCrawling Framework aims at providing instruments to configure and run your [Storm Crawler](http://stormcrawler.net/) based crawler. It mainly aims at easing crawling of article content publishing sites like news portals or blog sites. With the help of GUI tool Crawling Framework provides you can:\n\n1. Specify which sites to crawl.\n1. Configure URL inclusion and exclusion filters, thus controlling which sections of the site will be fetched.\n1. Specify which elements of the page provide information about article publication name, its title and main body.\n1. Define tests which validate that extraction rules are working.\n\nOnce configuration is done the Crawling Framework runs [Storm Crawler](http://stormcrawler.net/) based crawling following the rules specified in the configuration.\n\n## Introduction\n\nWe have recorded a video on how to setup and use Crawling Framework. Click on the image below to watch in on Youtube.\n\n[![Crawling Framework Intro](https://img.youtube.com/vi/AvO4lmmIuis/0.jpg)](https://www.youtube.com/watch?v=AvO4lmmIuis)\n\n## Requirements\n\nFramework writes its configuration and stores crawled data to ElasticSearch. Before starting crawl project [install ElasticSearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html) (Crawling Framework is tested to work with Elastic v7.x).\n\nCrawling Framework is a Java lib which will have to be extended to run Storm Crawler topology, thus Java (JDK8, Maven) infrastructure will be needed. \n\n### Using password protected ElasticSearch\n\nSome providers hide ElasticSearch under authentification step (Which makes sense). Just set environment variables `ES_USERNAME` and `ES_PASSWORD` accordingly, everything else can remain the same. Authentification step will be done implicitly if proper credentials are there\n\n## Configuring and Running a crawl\n\nSee [Crawling Framework Example](https://github.com/tokenmill/crawling-framework-example) project's documentation.\n\n\n## License\n\nCopyright \u0026copy; 2017-2019 [TokenMill UAB](http://www.tokenmill.ai).\n\nDistributed under the The Apache License, Version 2.0.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftokenmill%2Fcrawling-framework","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftokenmill%2Fcrawling-framework","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftokenmill%2Fcrawling-framework/lists"}