{"id":13548958,"url":"https://github.com/spatie/robots-txt","last_synced_at":"2025-05-14T15:04:26.472Z","repository":{"id":30208685,"uuid":"122979707","full_name":"spatie/robots-txt","owner":"spatie","description":"Determine if a page may be crawled from robots.txt, robots meta tags and robot headers","archived":false,"fork":false,"pushed_at":"2025-05-12T18:18:34.000Z","size":137,"stargazers_count":236,"open_issues_count":0,"forks_count":41,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-05-12T19:33:28.765Z","etag":null,"topics":["crawler","php","robots-txt"],"latest_commit_sha":null,"homepage":"https://spatie.be/en/opensource/php","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/spatie.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":"spatie","custom":"https://spatie.be/open-source/support-us"}},"created_at":"2018-02-26T14:15:52.000Z","updated_at":"2025-05-12T18:18:31.000Z","dependencies_parsed_at":"2024-12-22T08:01:53.041Z","dependency_job_id":"cbe018d8-ca86-4f45-9589-775772bedc6f","html_url":"https://github.com/spatie/robots-txt","commit_stats":{"total_commits":129,"total_committers":31,"mean_commits":4.161290322580645,"dds":0.7209302325581395,"last_synced_commit":"8e0cf015a5cb73f2699b5ffcfb1333d9ad927545"},"previous_names":[],"tags_count":25,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spatie%2Frobots-txt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spatie%2Frobots-txt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spatie%2Frobots-txt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spatie%2Frobots-txt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/spatie","download_url":"https://codeload.github.com/spatie/robots-txt/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254168703,"owners_count":22026206,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","php","robots-txt"],"created_at":"2024-08-01T12:01:16.492Z","updated_at":"2025-05-14T15:04:26.449Z","avatar_url":"https://github.com/spatie.png","language":"PHP","funding_links":["https://github.com/sponsors/spatie","https://spatie.be/open-source/support-us"],"categories":["PHP","Tools and Related Libraries"],"sub_categories":[],"readme":"# Parse `robots.txt`, `robots` meta and headers\n\n[![Latest Version on Packagist](https://img.shields.io/packagist/v/spatie/robots-txt.svg?style=flat-square)](https://packagist.org/packages/spatie/robots-txt)\n[![Tests](https://github.com/spatie/robots-txt/actions/workflows/run-tests.yml/badge.svg)](https://github.com/spatie/robots-txt/actions/workflows/run-tests.yml)\n[![Quality Score](https://img.shields.io/scrutinizer/g/spatie/robots-txt.svg?style=flat-square)](https://scrutinizer-ci.com/g/spatie/robots-txt)\n[![Total Downloads](https://img.shields.io/packagist/dt/spatie/robots-txt.svg?style=flat-square)](https://packagist.org/packages/spatie/robots-txt)\n\nDetermine if a page may be crawled from robots.txt, robots meta tags and robot headers.\n\n## Support us\n\n[\u003cimg src=\"https://github-ads.s3.eu-central-1.amazonaws.com/robots-txt.jpg?t=1\" width=\"419px\" /\u003e](https://spatie.be/github-ad-click/robots-txt)\n\nWe invest a lot of resources into creating [best in class open source packages](https://spatie.be/open-source). You can support us by [buying one of our paid products](https://spatie.be/open-source/support-us).\n\nWe highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using. You'll find our address on [our contact page](https://spatie.be/about-us). We publish all received postcards on [our virtual postcard wall](https://spatie.be/open-source/postcards).\n\n## Installation\n\nYou can install the package via composer:\n\n```bash\ncomposer require spatie/robots-txt\n```\n\n## Usage\n\n``` php\n$robots = Spatie\\Robots\\Robots::create();\n\n$robots-\u003emayIndex('https://www.spatie.be/nl/admin');\n\n$robots-\u003emayFollowOn('https://www.spatie.be/nl/admin');\n```\n\nYou can also specify a user agent:\n\n``` php\n$robots = Spatie\\Robots\\Robots::create('UserAgent007');\n```\n\nBy default, `Robots` will look for a `robots.txt` file on `https://host.com/robots.txt`.\nAnother location can be specified like so:\n\n``` php\n$robots = Spatie\\Robots\\Robots::create()\n    -\u003ewithTxt('https://www.spatie.be/robots-custom.txt');\n\n$robots = Spatie\\Robots\\Robots::create()\n    -\u003ewithTxt(__DIR__ . '/public/robots.txt');\n```\n\n### Testing\n\n``` bash\ncomposer test\n```\n\n### Changelog\n\nPlease see [CHANGELOG](CHANGELOG.md) for more information what has changed recently.\n\n## Contributing\n\nPlease see [CONTRIBUTING](https://github.com/spatie/.github/blob/main/CONTRIBUTING.md) for details.\n\n## Security Vulnerabilities\n\nPlease review [our security policy](../../security/policy) on how to report security vulnerabilities.\n\n## Postcardware\n\nYou're free to use this package, but if it makes it to your production environment we highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using.\n\nOur address is: Spatie, Kruikstraat 22, 2018 Antwerp, Belgium.\n\nWe publish all received postcards [on our company website](https://spatie.be/en/opensource/postcards).\n\n## Credits\n\n- [Brent Roose](https://github.com/brendt)\n- [All Contributors](../../contributors)\n\n## License\n\nThe MIT License (MIT). Please see [License File](LICENSE.md) for more information.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspatie%2Frobots-txt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspatie%2Frobots-txt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspatie%2Frobots-txt/lists"}