{"id":20704776,"url":"https://github.com/tekord/robots-txt-provider","last_synced_at":"2025-10-06T00:49:52.315Z","repository":{"id":57066482,"uuid":"374416674","full_name":"tekord/robots-txt-provider","owner":"tekord","description":"Provides various framework-agnostic ways to generate the contents of the robots.txt file","archived":false,"fork":false,"pushed_at":"2021-06-06T17:39:20.000Z","size":12,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-03-11T04:49:16.882Z","etag":null,"topics":["content-generation","framework-agnostic","php","robots-txt","seo","seo-optimization"],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tekord.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-06-06T17:00:38.000Z","updated_at":"2021-07-18T11:23:02.000Z","dependencies_parsed_at":"2022-08-24T07:50:14.439Z","dependency_job_id":null,"html_url":"https://github.com/tekord/robots-txt-provider","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/tekord/robots-txt-provider","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tekord%2Frobots-txt-provider","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tekord%2Frobots-txt-provider/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tekord%2Frobots-txt-provider/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tekord%2Frobots-txt-provider/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tekord","download_url":"https://codeload.github.com/tekord/robots-txt-provider/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tekord%2Frobots-txt-provider/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278542673,"owners_count":26004061,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-05T02:00:06.059Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["content-generation","framework-agnostic","php","robots-txt","seo","seo-optimization"],"created_at":"2024-11-17T01:14:40.286Z","updated_at":"2025-10-06T00:49:52.276Z","avatar_url":"https://github.com/tekord.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Framework-agnostic ROBOTS.TXT File Content Generator Kit\n\nThis package provides various framework-agnostic ways to generate the contents of the robots.txt file.\n\n## Installation\n\nInstall the package via Composer:\n\n```bash\ncomposer require tekord/robots-txt-provider\n```\n\n## Usage\n\nHere are the available content providers:\n\n- StringContentProvider - returns a plain string\n- FileContentProvider - returns the contents of the specified file\n- CallbackContentProvider - used a callback to get the value\n- CompositeContentProvider - used to compose multiple providers, returns the value of the first provider that returns a\n  non-NULL value\n\n### ContentBuilder\n\nThe ContentBuilder class provides handy methods for generating content:\n\n``` php\nContentBuilder::make()\n    -\u003eline(\"User-Agent: *\")\n    -\u003eemptyLine()\n    -\u003ecomment(\"This is a comment\")\n    -\u003eparameter(\"Host\", \"https://example.com\")\n    -\u003eallow(\"/about\")\n    -\u003edisallow(\"/login\")\n    -\u003ebuild();\n```\n\nWill return the following:\n\n```\nUser-Agent: *\n\n# This is a comment\nHost: https://example.com\nAllow: /about\nDisallow: /login\n```\n\nThe following methods have a conditional version: `line`, `emptyLine`, `comment`, `parameter`, `allow` and `disallow`.\nJust add `If` to the end of the method name like `lineIf`. The first parameter is a condition (boolean or callback). If\nthe condition is TRUE then the method executes. For instance:\n\n```php\n$debugMode = true;\n\n$useSpecialUserAgent = function () {\n    return false;\n};\n\nContentBuilder::make()\n    -\u003elineIf($useSpecialUserAgent, \"User-Agent: Bot/1.0\")\n    -\u003elineIf(!$useSpecialUserAgent, \"User-Agent: *\")\n    -\u003eemptyLine()\n    -\u003ecommentIf($debugMode, \"Debug mode is active\")\n    -\u003eparameter(\"Host\", \"https://example.com\")\n    -\u003edisallow(\"/\")\n    -\u003ebuild();\n```\n\nWill return the following:\n\n```\nUser-Agent: *\n\n# Debug mode is active\nHost: https://example.com\nDisallow: /\n```\n\n### StringContentProvider\n\n```php\n$content = \u003c\u003c\u003c'TXT'\nUser-Agent: *\nDisallow: /\nTXT;\n\n$contentProvider = new StringContentProvider($content);\n```\n\n### FileContentProvider\n\n```php\n$contentProvider = new FileContentProvider(__DIR__ . \"/storage/static/default-robots.txt\");\n```\n\n### CallbackContentProvider\n\n```php\n$contentProvider = new CallbackContentProvider(function() {\n    return ContentBuilder::make()\n        -\u003eline(\"User-Agent: *\")\n        -\u003eemptyLine()\n        -\u003ecomment(\"This content was generated by the CallbackContentProvider class\")\n        -\u003eparameter(\"Host\", \"https://example.com\")\n        -\u003edisallow(\"/\")\n        -\u003ebuild();\n});\n```\n\n### CompositeContentProvider\n\n```php\n$productionFileContentProvider = new FileContentProvider(__DIR__ . \"/public/robots.production.txt\");\n$defaultFileContentProvider = new FileContentProvider(__DIR__ . \"/public/robots.default.txt\");\n$fallbackFileContentProvider = new StringContentProvider(\"User-Agent: *\\nDisallow: /\");\n\n$compositeContentProvider = (new CompositeContentProvider())\n    -\u003eaddContentProvider($productionFileContentProvider)\n    -\u003eaddContentProvider($defaultFileContentProvider)\n    -\u003eaddContentProvider($fallbackFileContentProvider);\n```\n\n## Testing\n\n```bash\ncomposer test\n```\n\n## Security\n\nIf you discover any security related issues, please email [cyrill@tekord.space](mailto:cyrill@tekord.space) instead of\nusing the issue tracker.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftekord%2Frobots-txt-provider","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftekord%2Frobots-txt-provider","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftekord%2Frobots-txt-provider/lists"}