{"id":19238210,"url":"https://github.com/olical/crawlers","last_synced_at":"2026-06-29T19:31:06.526Z","repository":{"id":57713894,"uuid":"154303094","full_name":"Olical/crawlers","owner":"Olical","description":"Clojure(Script) library to identify crawler and bot user agent strings","archived":false,"fork":false,"pushed_at":"2018-10-23T10:51:40.000Z","size":14,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-07-01T23:17:52.569Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://clojars.org/olical/crawlers","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Olical.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-10-23T09:46:49.000Z","updated_at":"2023-07-25T14:20:53.000Z","dependencies_parsed_at":"2022-08-25T12:52:10.087Z","dependency_job_id":null,"html_url":"https://github.com/Olical/crawlers","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/Olical/crawlers","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Olical%2Fcrawlers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Olical%2Fcrawlers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Olical%2Fcrawlers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Olical%2Fcrawlers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Olical","download_url":"https://codeload.github.com/Olical/crawlers/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Olical%2Fcrawlers/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268385736,"owners_count":24242101,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-02T02:00:12.353Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T16:29:49.563Z","updated_at":"2026-06-29T19:31:06.483Z","avatar_url":"https://github.com/Olical.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Crawlers [![Clojars Project](https://img.shields.io/clojars/v/olical/crawlers.svg)](https://clojars.org/olical/crawlers)\n\nClojure(Script) library to identify crawler and bot user agent strings. Relies on [monperrus/crawler-user-agents][crawler-list-repo] for the regular expressions.\n\n## Usage\n\n```clojure\n(require '[crawlers.detector :as crawlers])\n\n;; This isn't a crawler.\n(time (crawlers/crawler? \"Mozilla/5.0 (X11; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0\"))\n;; \"Elapsed time: 0.214012 msecs\"\n;; =\u003e nil\n\n;; This is!\n(time (crawlers/crawler? \"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36\"))\n;; \"Elapsed time: 0.056963 msecs\"\n;; =\u003e \"Googlebot/\"\n```\n\n## Updating\n\nThe list of expressions is fetched from [`crawler-user-agents.json`][json-source] so it needs to be occasionally synchronised.\n\nYou can submit an update to the regular expression list by executing `clj -Afetch` and pasting the resulting vector in `src/crawlers/detector.cljc`.\n\nThe last update was performed at `Tue 23 Oct 11:46:14 BST 2018`. Pull requests to update this are welcome.\n\n## Unlicenced\n\nFind the full [unlicense][] in the `UNLICENSE` file, but here's a snippet.\n\n\u003eThis is free and unencumbered software released into the public domain.\n\u003e\n\u003eAnyone is free to copy, modify, publish, use, compile, sell, or distribute this software, either in source code form or as a compiled binary, for any purpose, commercial or non-commercial, and by any means.\n\nDo what you want. Learn as much as you can. Unlicense more software.\n\n[unlicense]: http://unlicense.org/\n[crawler-list-repo]: https://github.com/monperrus/crawler-user-agents\n[json-source]: https://raw.githubusercontent.com/monperrus/crawler-user-agents/master/crawler-user-agents.json\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Folical%2Fcrawlers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Folical%2Fcrawlers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Folical%2Fcrawlers/lists"}