{"id":18743287,"url":"https://github.com/rimiti/robotstxt","last_synced_at":"2025-07-27T10:18:12.393Z","repository":{"id":89493021,"uuid":"131134886","full_name":"rimiti/robotstxt","owner":"rimiti","description":"Robots.txt parser and generator - Work in progress","archived":false,"fork":false,"pushed_at":"2018-04-26T13:29:13.000Z","size":25,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-20T06:11:15.282Z","etag":null,"topics":["golang-package","robots-parser","robots-txt"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rimiti.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-04-26T09:46:40.000Z","updated_at":"2020-07-27T12:54:05.000Z","dependencies_parsed_at":null,"dependency_job_id":"93ba5321-b7e6-41e0-8cb7-17d381692615","html_url":"https://github.com/rimiti/robotstxt","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/rimiti/robotstxt","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rimiti%2Frobotstxt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rimiti%2Frobotstxt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rimiti%2Frobotstxt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rimiti%2Frobotstxt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rimiti","download_url":"https://codeload.github.com/rimiti/robotstxt/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rimiti%2Frobotstxt/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267342614,"owners_count":24071942,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-27T02:00:11.917Z","response_time":82,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang-package","robots-parser","robots-txt"],"created_at":"2024-11-07T16:10:55.563Z","updated_at":"2025-07-27T10:18:12.377Z","avatar_url":"https://github.com/rimiti.png","language":"Go","readme":"# robotstxt\n\nA robots.txt parser and generator written in Go, based on [samclarke robotstxt package](https://github.com/samclarke/robotstxt).\n\nIt currently supports:\n\n  * User-agent:\n  * Allow:\n  * Disallow:\n  * Sitemap:\n  * Crawl-delay:\n  * Host:\n  * URL encoded \u0026 UTF-8 paths\n  * Paths with wildcards (*) and EOL matching ($)\n\n## Installation\n\nGo get:\n\n    go get github.com/rimiti/robotstxt\n\n## Usage\n\n```go\nimport (\n    \"log\"\n    \"github.com/rimiti/robotstxt\"\n)\n\nfunc main() {\n    url := \"http://www.example.com/robots.txt\"\n    contents := `\n        User-agent: *\n        Disallow: /dir/\n        Disallow: /test.html\n        Allow: /dir/test.html\n        Allow: /test.html\n        Crawl-delay: 1\n        Sitemap: http://example.com/sitemap.xml\n        Host: example.com\n    `\n\n    robots, err := Parse(contents, url)\n    if err != nil {\n        log.Fatalln(err.Error())\n    }\n\n    allowed, _ := robots.IsAllowed(\"Bot/1.0\", \"http://www.example.com/test.html\")\n    if !allowed {\n        println(\"Not allowed to crawl: /test.html\")\n    }\n\n    allowed, _ = robots.IsAllowed(\"Bot/1.0\", \"http://www.example.com/dir/test.html\")\n    if allowed {\n        println(\"Allowed to crawl: /dir/test.html\")\n    }\n\n    // 1\n    println(\"Crawl delay: \" + robots.CrawlDelay(\"Bot/1.0\"))\n\n    // [http://example.com/sitemap.xml]\n    println(\"Sitemaps: \" + strings.Join(robots.Sitemaps(), \",\"))\n\n    // example.com\n    println(\"Preferred host: \" + robots.Host())\n}\n```","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frimiti%2Frobotstxt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frimiti%2Frobotstxt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frimiti%2Frobotstxt/lists"}