{"id":37022552,"url":"https://github.com/itechbear/robotstxt","last_synced_at":"2026-01-14T02:41:46.867Z","repository":{"id":57720287,"uuid":"243689872","full_name":"itechbear/robotstxt","owner":"itechbear","description":"A java clone of Google's robotst.txt parser: https://github.com/google/robotstxt","archived":false,"fork":false,"pushed_at":"2020-10-25T09:08:00.000Z","size":44,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2023-09-24T14:35:44.430Z","etag":null,"topics":["crawler","google-robotst-parser","java","robotstxt"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/itechbear.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-02-28T06:00:39.000Z","updated_at":"2020-10-25T09:08:02.000Z","dependencies_parsed_at":"2022-09-26T21:41:20.634Z","dependency_job_id":null,"html_url":"https://github.com/itechbear/robotstxt","commit_stats":null,"previous_names":[],"tags_count":1,"template":null,"template_full_name":null,"purl":"pkg:github/itechbear/robotstxt","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itechbear%2Frobotstxt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itechbear%2Frobotstxt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itechbear%2Frobotstxt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itechbear%2Frobotstxt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/itechbear","download_url":"https://codeload.github.com/itechbear/robotstxt/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itechbear%2Frobotstxt/sbom","scorecard":{"id":496907,"data":{"date":"2025-08-11","repo":{"name":"github.com/itechbear/robotstxt","commit":"7b9b1817485eb63cf297b17c4965a85de8d5eced"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3,"checks":[{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Code-Review","score":0,"reason":"Found 0/7 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}}]},"last_synced_at":"2025-08-19T20:39:01.593Z","repository_id":57720287,"created_at":"2025-08-19T20:39:01.593Z","updated_at":"2025-08-19T20:39:01.593Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28408712,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T01:52:23.358Z","status":"online","status_checked_at":"2026-01-14T02:00:06.678Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","google-robotst-parser","java","robotstxt"],"created_at":"2026-01-14T02:41:46.153Z","updated_at":"2026-01-14T02:41:46.853Z","avatar_url":"https://github.com/itechbear.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# robotstxt (java)\n\nA java clone of [Google's robotst.txt parser](https://github.com/google/robotstxt), passing all unit tests.\n\n**Disclaimer:**\n- The author of this repository is not affiliated to Google by any means.\n\n# Features\n\n- With google specific optimizations, compared with other implmentations (credit to Google authors)\n- No extra dependency other than JDK\n\n# How to use\n\n- Add this package to your build dependency, based on your build tool:\n  - Maven\n    ```xml\n    \u003cdependency\u003e\n      \u003cgroupId\u003ecom.github.itechbear\u003c/groupId\u003e\n      \u003cartifactId\u003erobotstxt\u003c/artifactId\u003e\n      \u003cversion\u003e0.0.1\u003c/version\u003e\n    \u003c/dependency\u003e\n    ```\n  - Gradle(Groovy)\n    ```groovy\n    implementation 'com.github.itechbear:robotstxt:0.0.1'\n    ```\n  - SBT\n    ```scale\n    libraryDependencies += \"com.github.itechbear\" % \"robotstxt\" % \"0.0.1\"\n    ```\n  - For any other build tool, please refer to [https://search.maven.org/artifact/com.github.itechbear/robotstxt/0.0.1/jar](https://search.maven.org/artifact/com.github.itechbear/robotstxt/0.0.1/jar)\n\n- Code sample\n```java\nString robotstxt = \"allow: /foo/bar/\\n\" +\n                \"\\n\" +\n                \"user-agent: FooBot\\n\" +\n                \"disallow: /\\n\" +\n                \"allow: /x/\\n\" +\n                \"user-agent: BarBot\\n\" +\n                \"disallow: /\\n\" +\n                \"allow: /y/\\n\" +\n                \"\\n\" +\n                \"\\n\" +\n                \"allow: /w/\\n\" +\n                \"user-agent: BazBot\\n\" +\n                \"\\n\" +\n                \"user-agent: FooBot\\n\" +\n                \"allow: /z/\\n\" +\n                \"disallow: /\\n\";\nString url = \"http://test.com/x\";\n\nRobotsMatcher matcher = new RobotsMatcher();\n// check whether FooBot is allowed to crawl url.\nmatcher.OneAgentAllowedByRobots(robotstxt, \"FooBot\", url);\n// check whether any of (FooBot,BarBot) is allowed to crawl url\nmatcher.AllowedByRobots(robotstxt, Arrays.asList(\"FooBot\", \"BarBot\"), url);\n```\n\n# Change log\n\n- 0.0.1 Initial release, based on [google/robotstxt@750aec7](https://github.com/google/robotstxt/tree/750aec7933648c816d6d5bb2f4fe5c30f2485ccf)  \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fitechbear%2Frobotstxt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fitechbear%2Frobotstxt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fitechbear%2Frobotstxt/lists"}