{"id":13441077,"url":"https://github.com/google/robotstxt","last_synced_at":"2025-05-14T19:05:38.108Z","repository":{"id":34989103,"uuid":"191419070","full_name":"google/robotstxt","owner":"google","description":"The repository contains Google's robots.txt parser and matcher as a C++ library (compliant to C++11).","archived":false,"fork":false,"pushed_at":"2024-08-02T04:59:33.000Z","size":125,"stargazers_count":3411,"open_issues_count":16,"forks_count":237,"subscribers_count":95,"default_branch":"master","last_synced_at":"2025-04-11T06:17:44.967Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-06-11T17:30:19.000Z","updated_at":"2025-04-07T23:47:16.000Z","dependencies_parsed_at":"2024-01-10T00:00:45.143Z","dependency_job_id":"a03f28d7-2811-4454-935d-5111d1212e35","html_url":"https://github.com/google/robotstxt","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Frobotstxt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Frobotstxt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Frobotstxt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google%2Frobotstxt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google","download_url":"https://codeload.github.com/google/robotstxt/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248351394,"owners_count":21089272,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T03:01:29.669Z","updated_at":"2025-04-11T06:17:49.062Z","avatar_url":"https://github.com/google.png","language":"C++","funding_links":[],"categories":["C++","\u003ca id=\"1d9dec1320a5d774dc8e0e7604edfcd3\"\u003e\u003c/a\u003e工具-新添加的"],"sub_categories":["\u003ca id=\"8f1b9c5c2737493524809684b934d49a\"\u003e\u003c/a\u003e文章\u0026\u0026视频"],"readme":"\n# Google Robots.txt Parser and Matcher Library\n\nThe repository contains Google's robots.txt parser and matcher as a C++ library\n(compliant to C++14).\n\n## About the library\n\nThe Robots Exclusion Protocol (REP) is a standard that enables website owners to\ncontrol which URLs may be accessed by automated clients (i.e. crawlers) through\na simple text file with a specific syntax. It's one of the basic building blocks\nof the internet as we know it and what allows search engines to operate.\n\nBecause the REP was only a de-facto standard for the past 25 years, different\nimplementers implement parsing of robots.txt slightly differently, leading to\nconfusion. This project aims to fix that by releasing the parser that Google\nuses.\n\nThe library is slightly modified (i.e. some internal headers and equivalent\nsymbols) production code used by Googlebot, Google's crawler, to determine which\nURLs it may access based on rules provided by webmasters in robots.txt files.\nThe library is released open-source to help developers build tools that better\nreflect Google's robots.txt parsing and matching.\n\nFor webmasters, we included a small binary in the project that allows testing a\nsingle URL and user-agent against a robots.txt.\n\n## Building the library\n\n### Quickstart\n\nWe included with the library a small binary to test a local robots.txt against a\nuser-agent and URL. Running the included binary requires:\n\n*   A compatible platform (e.g. Windows, macOS, Linux, etc.). Most platforms are\n    fully supported.\n*   A compatible C++ compiler supporting at least C++14. Most major compilers\n    are supported.\n*   [Git](https://git-scm.com/) for interacting with the source code repository.\n    To install Git, consult the\n    [Set Up Git](https://help.github.com/articles/set-up-git/) guide on\n    [GitHub](https://github.com/).\n*   Although you are free to use your own build system, most of the\n    documentation within this guide will assume you are using\n    [Bazel](https://bazel.build/). To download and install Bazel (and any of its\n    dependencies), consult the\n    [Bazel Installation Guide](https://docs.bazel.build/versions/master/install.html)\n\n#### Building with Bazel\n\n[Bazel](https://bazel.build/) is the official build system for the library,\nwhich is supported on most major platforms (Linux, Windows, MacOS, for example)\nand compilers.\n\nTo build and run the binary:\n\n```bash\n$ git clone https://github.com/google/robotstxt.git robotstxt\nCloning into 'robotstxt'...\n...\n$ cd robotstxt/\nbazel-robots$ bazel test :robots_test\n...\n/:robots_test                                                      PASSED in 0.1s\n\nExecuted 1 out of 1 test: 1 test passes.\n...\nbazel-robots$ bazel build :robots_main\n...\nTarget //:robots_main up-to-date:\n  bazel-bin/robots_main\n...\nbazel-robots$ bazel run robots_main -- ~/local/path/to/robots.txt YourBot https://example.com/url\n  user-agent 'YourBot' with url 'https://example.com/url' allowed: YES\n```\n\n#### Building with CMake\n\n[CMake](https://cmake.org) is the community-supported build system for the\nlibrary.\n\nTo build the library using CMake, just follow the steps below:\n\n```bash\n$ git clone https://github.com/google/robotstxt.git robotstxt\nCloning into 'robotstxt'...\n...\n$ cd robotstxt/\n...\n$ mkdir c-build \u0026\u0026 cd c-build\n...\n$ cmake .. -DROBOTS_BUILD_TESTS=ON\n...\n$ make\n...\n$ make test\nRunning tests...\nTest project robotstxt/c-build\n    Start 1: robots-test\n1/1 Test #1: robots-test ......................   Passed    0.02 sec\n\n100% tests passed, 0 tests failed out of 1\n\nTotal Test time (real) =   0.02 sec\n...\n$ robots ~/local/path/to/robots.txt YourBot https://example.com/url\n  user-agent 'YourBot' with url 'https://example.com/url' allowed: YES\n```\n\n## Notes\n\nParsing of robots.txt files themselves is done exactly as in the production\nversion of Googlebot, including how percent codes and unicode characters in\npatterns are handled. The user must ensure however that the URI passed to the\nAllowedByRobots and OneAgentAllowedByRobots functions, or to the URI parameter\nof the robots tool, follows the format specified by RFC3986, since this library\nwill not perform full normalization of those URI parameters. Only if the URI is\nin this format, the matching will be done according to the REP specification.\n\nAlso note that the library, and the included binary, do not handle\nimplementation logic that a crawler might apply outside of parsing and matching,\nfor example: `Googlebot-Image` respecting the rules specified for `User-agent:\nGooglebot` if not explicitly defined in the robots.txt file being tested.\n\n## License\n\nThe robots.txt parser and matcher C++ library is licensed under the terms of the\nApache license. See LICENSE for more information.\n\n## Links\n\nTo learn more about this project:\n\n*   check out the\n    [Robots Exclusion Protocol standard](https://www.rfc-editor.org/rfc/rfc9309.html),\n*   how\n    [Google Handles robots.txt](https://developers.google.com/search/reference/robots_txt),\n*   or for a high level overview, the\n    [robots.txt page on Wikipedia](https://en.wikipedia.org/wiki/Robots_exclusion_standard).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle%2Frobotstxt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle%2Frobotstxt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle%2Frobotstxt/lists"}