{"id":20836306,"url":"https://github.com/igrigorik/canicrawl","last_synced_at":"2025-05-08T02:38:25.265Z","repository":{"id":66004963,"uuid":"1752440","full_name":"igrigorik/canicrawl","owner":"igrigorik","description":"Hosted robots.txt permissions verifier","archived":false,"fork":false,"pushed_at":"2014-06-07T18:09:31.000Z","size":137,"stargazers_count":23,"open_issues_count":0,"forks_count":5,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-05-08T18:35:47.509Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://canicrawl.appspot.com","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/igrigorik.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2011-05-15T21:02:59.000Z","updated_at":"2019-07-12T11:38:31.000Z","dependencies_parsed_at":"2023-02-19T21:15:30.921Z","dependency_job_id":null,"html_url":"https://github.com/igrigorik/canicrawl","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/igrigorik%2Fcanicrawl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/igrigorik%2Fcanicrawl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/igrigorik%2Fcanicrawl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/igrigorik%2Fcanicrawl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/igrigorik","download_url":"https://codeload.github.com/igrigorik/canicrawl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252988224,"owners_count":21836475,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-18T00:29:28.750Z","updated_at":"2025-05-08T02:38:25.233Z","avatar_url":"https://github.com/igrigorik.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Can I Crawl (this URL)\n\nHosted robots.txt permissions verifier.\n\n## ENDPOINTS\n\n- [`/`](http://canicrawl.appspot.com/) This page.\n- [`/check`](http://canicrawl.appspot.com/check) Runs the robots.txt verification check.\n\n## Description\n\nVerifies if the provided URL is allowed to be crawled by your User-Agent. Pass in the destination URL and the service will download, parse and check the [robots.txt](http://www.robotstxt.org/) file for permissions. If you're allowed to continue, it will issue a **3XX** redirect, otherwise a **4XX** code is returned.\n\n## Examples\n\n### $ curl -v http://canicrawl.appspot.com/check?url=http://google.com/\n\t\u003c HTTP/1.0 302 Found\n\t\u003c Location: http://www.google.com/\n\n### $ curl -v http://canicrawl.appspot.com/check?url=http://google.com/search\n\t\u003c HTTP/1.0 403 Forbidden\n\t\u003c Content-Length: 23\n\t{\"status\":\"disallowed\"}\n\n### $ curl -H'User-Agent: MyCustomAgent' -v http://canicrawl.appspot.com/check?url=http://google.com/\n\t\u003e User-Agent: MyCustomAgent\n\t\u003c HTTP/1.0 302 Found\n\t\u003c Location: http://www.google.com/\n\nNote: [google.com/robots.txt](http://google.com/robots.txt) disallows requests to _/search_.\n\n### License\n\nMIT License - Copyright (c) 2011 [Ilya Grigorik](http://www.igvita.com/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Figrigorik%2Fcanicrawl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Figrigorik%2Fcanicrawl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Figrigorik%2Fcanicrawl/lists"}