{"id":50897832,"url":"https://github.com/brianmadden/krawler","last_synced_at":"2026-07-22T07:01:13.871Z","repository":{"id":49100083,"uuid":"71804596","full_name":"brianmadden/krawler","owner":"brianmadden","description":"A web crawling framework written in Kotlin","archived":false,"fork":false,"pushed_at":"2021-06-29T00:31:08.000Z","size":413,"stargazers_count":133,"open_issues_count":6,"forks_count":16,"subscribers_count":6,"default_branch":"master","last_synced_at":"2026-06-13T20:34:34.411Z","etag":null,"topics":["crawler4j","framework","kotlin","link-checker","web-crawler","web-crawling","webcrawler"],"latest_commit_sha":null,"homepage":"","language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/brianmadden.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-10-24T15:44:27.000Z","updated_at":"2026-05-16T08:19:35.000Z","dependencies_parsed_at":"2022-09-14T11:31:35.992Z","dependency_job_id":null,"html_url":"https://github.com/brianmadden/krawler","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/brianmadden/krawler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brianmadden%2Fkrawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brianmadden%2Fkrawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brianmadden%2Fkrawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brianmadden%2Fkrawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/brianmadden","download_url":"https://codeload.github.com/brianmadden/krawler/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brianmadden%2Fkrawler/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35751644,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-22T02:00:06.236Z","response_time":124,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler4j","framework","kotlin","link-checker","web-crawler","web-crawling","webcrawler"],"created_at":"2026-06-16T01:31:30.082Z","updated_at":"2026-07-22T07:01:13.866Z","avatar_url":"https://github.com/brianmadden.png","language":"Kotlin","funding_links":[],"categories":["framework"],"sub_categories":[],"readme":"[![Release](https://jitpack.io/v/brianmadden/krawler.svg)](https://jitpack.io/#brianmadden/krawler) \n[![Build Status](https://travis-ci.org/brianmadden/krawler.svg?branch=master)](https://travis-ci.org/brianmadden/krawler)\n[![Awesome Kotlin Badge](https://kotlin.link/awesome-kotlin.svg)](https://github.com/KotlinBy/awesome-kotlin)\n\nAbout\n=====\n\nKrawler is a web crawling framework written in Kotlin. It is heavily inspired by\n[crawler4j](https://github.com/yasserg/crawler4j) by Yasser Ganjisaffar. The project \nis still very new, and those looking for a mature, well tested crawler framework should\nlikely still use crawler4j. For those who can tolerate a bit of turbulence, Krawler should serve as\na replacement for crawler4j with minimal modifications to existing applications.\n \nSome neat features and benefits of Krawler include:\n\n* Kotlin project!\n* Krawler differentiates between a \"check\" and a \"visit\". \nChecks are used to verify the status code of a resource by issuing an HTTP HEAD request rather than a GET request.\nEach policy (get or check) can have it's own logic associated with it by implementing \neither `shouldCheck` or `shouldVisit` and `check` and `visit`.\n* Krawler's politeness delay is per-host rather than global. This way servers aren't overwhelmed, but crawls visiting\nmany hosts in parallel are not effectively serialized by the politeness delay.\n* Krawler uses Jsoup for parsing HTML files while harvesting links, making it more tolerant of malformed or \npoorly written websites, and thus less likely to error out during a crawl. The original HTML of the page is\nstill available to facilitate validation and checking though.\n* Krawler collects full anchor tags including all attributes and anchor text.\n* Krawler currently has no proxy support, but it is on the roadmap. :(\n\nAdd Dependency\n======\nKrawler is published through jitpack.io at: https://jitpack.io/#brianmadden/krawler/ . \nAdd jitpack.io as a repository, and krawler as a dependency to use Krawler in your project:\n\n#### Using Gradle\n```gradle\nrepositories {\n    jcenter()\n    maven { url \"https://jitpack.io\" }\n}\n\ndependencies {\n    compile 'com.github.brianmadden:krawler:0.4.4'\n}\n```\n#### Using Maven\n```xml\n\u003crepositories\u003e\n    \u003crepository\u003e\n        \u003cid\u003ejitpack.io\u003c/id\u003e\n        \u003curl\u003ehttps://jitpack.io\u003c/url\u003e\n    \u003c/repository\u003e\n\u003c/repositories\u003e\n\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.github.brianmadden\u003c/groupId\u003e\n    \u003cartifactId\u003ekrawler\u003c/artifactId\u003e\n    \u003cversion\u003e0.4.4\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nUsage\n=====\nUsing the Krawler framework is fairly simple. Minimally, there are two methods that must be overridden\nin order to use the framework. Overriding the `shouldVisit` method dictates what should be visited by\nthe crawler, and the `visit` method dictates what happens once the page is visited. Overriding these\ntwo methods is sufficient for creating your own crawler, however there are additional methods that\ncan be overridden to privde more robust behavior.\n\nThe full code for this simple example can also be found in the [example project](...):\n```kotlin\nclass SimpleExample(config: KrawlConfig = KrawlConfig()) : Krawler(config) {\n\n    private val FILTERS: Regex = Regex(\".*(\\\\.(css|js|bmp|gif|jpe?g|png|tiff?|mid|mp2|mp3|mp4|wav|avi|\" +\n            \"mov|mpeg|ram|m4v|pdf|rm|smil|wmv|swf|wma|zip|rar|gz|tar|ico))$\", RegexOption.IGNORE_CASE)\n\n    /**\n     * Threadsafe whitelist of acceptable hosts to visit\n     */\n    val whitelist: MutableSet\u003cString\u003e = ConcurrentSkipListSet()\n\n    override fun shouldVisit(url: KrawlUrl): Boolean {\n        val withoutGetParams: String = url.canonicalForm.split(\"?\").first()\n        return (!FILTERS.matches(withoutGetParams) \u0026\u0026 url.host in whitelist)\n    }\n\n    private val counter: AtomicInteger = AtomicInteger(0)\n\n    override fun visit(url: KrawlUrl, doc: KrawlDocument) {\n        println(\"${counter.incrementAndGet()}. Crawling ${url.canonicalForm}\")\n    }\n\n    override fun onContentFetchError(url: KrawlUrl, reason: String) {\n        println(\"${counter.incrementAndGet()}. Tried to crawl ${url.canonicalForm} but failed to read the content.\")\n    }\n\n    private var startTimestamp: Long = 0\n    private var endTimestamp: Long = 0\n\n    override fun onCrawlStart() {\n        startTimestamp = LocalTime.now().toNanoOfDay()\n    }\n    override fun onCrawlEnd() {\n        endTimestamp = LocalTime.now().toNanoOfDay()\n        println(\"Crawled $counter pages in ${(endTimestamp - startTimestamp) / 1000000000.0} seconds.\")\n    }\n}\n```\n\nRoadmap\n=======\n* Proxy support\n* Headless Chrome support for crawling Javascript driven sites\n\nRelease Notes\n=============\n**0.4.4 (2020-1-29)**\n- Upgrade Kotlin to 1.3.61\n- Upgrade `kotlinx.coroutines`. This required an update to some of the places where coroutine builders were called internally.\n- Upgrade Gradle wrapper\n\n**0.4.3 (2017-11-20)**\n- Added ability to clear crawl queues by RequestId and Age, see `Krawler#removeUrlsByRootPage` \n  and `Krawler#removeUrlsByAge`\n- Added config option to prevent crawler shutdown on empty queues\n- Added new single byte priority field to `KrawlQueueEntry`. Queues will always attempt to pop the `lowest` priority\n  entry available. Priority can be assigned by overriding the `Krawler#assignQueuePriorty` method.\n- Update dependencies\n\n**0.4.2 (2017-10-25)**\n- Updated to Kotlin Runtime 1.1.51, kotlinx-coroutines 0.19.2\n- Reworked KrawlUrl class internals to handle spaces in URLs better which should result in\nmore stability when crawling.\n\n**0.4.1 (2017-8-15)**\n- Removed logging implementation from dependencies to prevent logging conflicts when used as a library.\n- Updated Kotlin version to 1.1.4\n- Updated `kotlinx.coroutines` to .17\n\n**0.4.0 (2017-5-17)**\n- Rewrote core crawl loop to use Kotlin 1.1 coroutines. This has effectively turned the crawl process into a\nmulti-stage pipeline. This architecture change has removed the necessity for some locking by removing resource \ncontention by multiple threads.\n\n- Updated the build file to build the simple example as a runnable jar\n \n- Minor bug fixes in the KrawlUrl class.\n\n**0.3.2 (2017-3-3)**\n- Fixed a number of bugs that would result in a crashed thread, and subsequently an incorrect number of crawled pages\nas well as cause slowdowns due to a reduced number of worker threads.\n\n- Added a new utility function to wrap `doCrawl` and log any uncaught exceptions during crawling. \n\n**0.3.1 (2017-2-2)**\n- Created 1:1 mapping between threads and the number of queues used to serve URLs to visit. URLs have an\naffinity for a particular queue based on their domain. All URLs from that domain will end up in the same\nqueue. This improves parallel crawl performance by reducing the frequency that the politeness delay\neffects requests. For crawls bound to fewer domains than queues, the excess queues are not used.\n\n- Many bug fixes including fix that eliminates accidental over-crawling.\n\n**0.2.2 (2017-1-21)**\n- Added additional configuration option for redirect handling in KrawlConfig. Setting\n`useFastRedirectHandling = true` (when redirects are enabled) will cause Krawler to \nautomatically follow redirects, keeping a history of the transitions and status codes.\nThis history is present in the `KrawlDocument#redirectHistory` property.\n\n\n**0.2.1 (2017-1-20)**\n- Redirect handling has been changed. Redirects can be followed or not via configuration\noption in `KrawlConfig`. When redirects are enabled the redirected to URL will be added \nto the queue as a part of the link harvesting phase of Krawler.\n\n- If an anchor tag specifies `rel='canonical'` the `canonicalForm` will not be subject\nto further processing.\n\n- `KrawlUrl.new`'s implementation has been changed to prevent `null` from being returned\nin certain circumstances.\n\n\n**0.2.0 (2017-1-18)** \n\n- Krawler now respects robots.txt. This feature can be configured by passing a custom `RobotsConfig` \nto your `Krawler` instance. By default Krawler will respect robots.txt without any additional configuration.\n- Krawler now collects outgoing links from `src` attributes of tags in addition to the `href` of anchor tags.\n- Minor bug fixes and refactorings.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrianmadden%2Fkrawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrianmadden%2Fkrawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrianmadden%2Fkrawler/lists"}