{"id":24778350,"url":"https://github.com/larevanchedessites/google-robotstxt-ruby","last_synced_at":"2025-08-17T11:04:56.074Z","repository":{"id":56874954,"uuid":"224836467","full_name":"larevanchedessites/google-robotstxt-ruby","owner":"larevanchedessites","description":"🤖 Ruby gem wrapper around Google Robotstxt Parser C++ library","archived":false,"fork":false,"pushed_at":"2019-12-06T15:20:23.000Z","size":19,"stargazers_count":4,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-08-13T09:52:49.870Z","etag":null,"topics":["c-plus-plus","cpp","gem","google","robots-parser","robots-txt","ruby","ruby-gem","rubygem","rubygems","seo"],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/larevanchedessites.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-11-29T10:57:07.000Z","updated_at":"2022-12-28T15:43:52.000Z","dependencies_parsed_at":"2022-08-20T10:11:13.306Z","dependency_job_id":null,"html_url":"https://github.com/larevanchedessites/google-robotstxt-ruby","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/larevanchedessites/google-robotstxt-ruby","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/larevanchedessites%2Fgoogle-robotstxt-ruby","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/larevanchedessites%2Fgoogle-robotstxt-ruby/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/larevanchedessites%2Fgoogle-robotstxt-ruby/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/larevanchedessites%2Fgoogle-robotstxt-ruby/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/larevanchedessites","download_url":"https://codeload.github.com/larevanchedessites/google-robotstxt-ruby/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/larevanchedessites%2Fgoogle-robotstxt-ruby/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270837583,"owners_count":24654391,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-17T02:00:09.016Z","response_time":129,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["c-plus-plus","cpp","gem","google","robots-parser","robots-txt","ruby","ruby-gem","rubygem","rubygems","seo"],"created_at":"2025-01-29T08:19:15.088Z","updated_at":"2025-08-17T11:04:56.031Z","avatar_url":"https://github.com/larevanchedessites.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"NOTE: This repo is still under construction, please use at your own risk. :-)\n\n# Google Robotstxt Parser C++ Library Wrapper Gem\nThis is a unofficial Ruby gem that provides a wrapper around [Google Robotstxt Parser C++ library](https://github.com/google/robotstxt).\n\n## Installation\nAdd this line to your application's Gemfile:\n```\ngem 'google_robotstxt_parser'\n```\n\nAnd then execute:\n```\nbundle\n```\n\nOr install it yourself as:\n```\ngem install google_robotstxt_parser\n```\n\n## Usage\nAt the time there is only one method available which is demonstrated below:\n```\nrequire './robotstxt'\ninclude Robotstxt\n\nrobotstxt_content = \"# robotstxt.org/\\n\\nUser-agent: *\\nDisallow: /\\n\\nSitemap: https://www.bqst.fr/sitemap.xml\"\nuser_agent = \"GoogleBot\"\nurl = \"https://www.bqst.fr\"\n\nallowed_by_robots(robotstxt_content, user_agent, url)\n# return true if user_agent is allowed to access url\n```\n\n## Deploy\nIf you're using [Heroku](https://www.heroku.com), you'll need [CMAKE buildpacks](https://elements.heroku.com/buildpacks/Starkast/heroku-buildpack-cmake) to build this gems. To add it to your app :\n\n```\n$ heroku buildpacks:remove heroku/ruby\n$ heroku buildpacks:add starkast/cmake\n$ heroku buildpacks:add heroku/ruby\n$ heroku buildpacks \n=== mytool Buildpack URLs\n1. starkast/cmake\n2. heroku/ruby\n$ git push heroku master\n```\n\n## Todo\nThere are quite a few outstanding tasks:\n- [ ] Some tests :-)\n\n\n## Author\n[Bastien Montois](https://www.bqst.fr) @ [la-revanche-des-sites.fr](https://www.la-revanche-des-sites.fr)\n\n## Contributing\nBug reports and pull requests are welcome on GitHub at https://github.com/larevanchedessites/google-robotstxt-ruby. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.\n\n## History\nView the [changelog](https://github.com/larevanchedessites/google-robotstxt-ruby/blob/master/CHANGELOG.md). This gem follows [Semantic Versioning](http://semver.org/).\n\n## License\nThe gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).\nSee [LICENSE](https://github.com/larevanchedessites/google-robotstxt-ruby/blob/master/LICENSE) for details.\n\n## Code of Conduct\nEveryone interacting in the project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/larevanchedessites/google-robotstxt-ruby/blob/master/CODE_OF_CONDUCT.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flarevanchedessites%2Fgoogle-robotstxt-ruby","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flarevanchedessites%2Fgoogle-robotstxt-ruby","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flarevanchedessites%2Fgoogle-robotstxt-ruby/lists"}