{"id":22305697,"url":"https://github.com/muratgozel/robotstxt-util","last_synced_at":"2025-07-29T04:32:39.675Z","repository":{"id":98551560,"uuid":"265245982","full_name":"muratgozel/robotstxt-util","owner":"muratgozel","description":"RFC 9309 spec compliant robots.txt builder and parser. 🦾 No dependencies, fully typed.","archived":false,"fork":false,"pushed_at":"2024-09-14T08:46:08.000Z","size":139,"stargazers_count":3,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-11-10T09:14:29.278Z","etag":null,"topics":["rfc-5234","robots-builder","robots-exclusion-protocol","robots-generator","robots-parser","robots-txt"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/muratgozel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null},"funding":{"github":null,"patreon":"muratgozel","open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"custom":null}},"created_at":"2020-05-19T12:49:53.000Z","updated_at":"2024-09-14T08:45:50.000Z","dependencies_parsed_at":"2023-06-09T14:45:13.408Z","dependency_job_id":null,"html_url":"https://github.com/muratgozel/robotstxt-util","commit_stats":{"total_commits":3,"total_committers":1,"mean_commits":3.0,"dds":0.0,"last_synced_commit":"0710ed32345e710cec8283fdfea64dbfd9194cb0"},"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muratgozel%2Frobotstxt-util","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muratgozel%2Frobotstxt-util/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muratgozel%2Frobotstxt-util/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/muratgozel%2Frobotstxt-util/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/muratgozel","download_url":"https://codeload.github.com/muratgozel/robotstxt-util/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227981888,"owners_count":17850920,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["rfc-5234","robots-builder","robots-exclusion-protocol","robots-generator","robots-parser","robots-txt"],"created_at":"2024-12-03T19:12:48.539Z","updated_at":"2024-12-03T19:12:49.233Z","avatar_url":"https://github.com/muratgozel.png","language":"TypeScript","funding_links":["https://patreon.com/muratgozel"],"categories":[],"sub_categories":[],"readme":"# robotstxt-util\nRFC 9309 spec compliant robots.txt builder and parser. 🦾 No dependencies, fully typed.\n\n![NPM](https://img.shields.io/npm/l/robotstxt-util)\n[![Build status](https://badge.buildkite.com/59019ef6df1cc44bcb5b790bd21f198d1e488c842624c62cd8.svg)](https://buildkite.com/gozel/robotstxt-util)\n\nBefore using this library, I recommend you to read the following guide by Google:\n[https://developers.google.com/search/docs/crawling-indexing/robots/intro](https://developers.google.com/search/docs/crawling-indexing/robots/intro)\n\nNote to myself (and contributors):\n[https://www.rfc-editor.org/rfc/rfc9309.html](https://www.rfc-editor.org/rfc/rfc9309.html)\n\n## Install\n```sh\nnpm i robotstxt-util\n```\n\n## Use\nExports a parser `parseRobotsTxt` and an object `RobotsTxt` to create and manage robots.txt data.\n\n### Create robots.txt\n```js\nimport { RobotsTxt } from 'robotstxt-util'\n\nconst robotstxt = new RobotsTxt()\n\nconst allBots = robotstxt.newGroup('*')\nallBots.disallow('/')\n\nconst googleBot = robotstxt.newGroup('googlebot')\ngoogleBot.allow('/abc')\ngoogleBot.disallow('/def').disallow('/jkl')\n\n// specify multiple bots\nconst otherBots = robotstxt.newGroup(['abot', 'bbot', 'cbot'])\ngoogleBot.allow('/qwe')\n// specify custom rules\ngoogleBot.addCustomRule('crawl-delay', 10)\n\n// add sitemaps\nrobotstxt.add('sitemap', 'https://yoursite/sitemap.en.xml')\nrobotstxt.add('sitemap', 'https://yoursite/sitemap.tr.xml')\n\n// and export\nconst json = robotstxt.json()\nconst txt = robotstxt.txt()\n```\n\n### Parse robots.txt data\nParses the data and returns instance of `RobotsTxt`:\n```js\nimport { parseRobotsTxt } from 'robotstxt-util'\n\nconst data = `\n# hello robots\n\nUser-Agent: *\nDisallow: *.gif$\nDisallow: /example/\nAllow: /publications/\n\nUser-Agent: foobot\nDisallow:/\ncrawl-delay: 10\nAllow:/example/page.html\nAllow:/example/allowed.gif\n\n# comments will be stripped out\n\nUser-Agent: barbot\nUser-Agent: bazbot\nDisallow: /example/page.html\n\nSitemap: https://yoursite/sitemap.en.xml\nSitemap: https://yoursite/sitemap.tr.xml\n`\nconst robotstxt = parseRobotsTxt(data)\n\n// update something in some group\nrobotstxt.findGroup('barbot').allow('/aaa').allow('/bbb')\n\n// store as json or do whatever you want\nconst json = robotstxt.json()\n```\n\n## Contributing\nIf you're interested in contributing, read the [CONTRIBUTING.md](https://github.com/muratgozel/muratgozel/blob/main/CONTRIBUTING.md) first, please.\n\n---\n\nThanks for watching 🐬\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmuratgozel%2Frobotstxt-util","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmuratgozel%2Frobotstxt-util","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmuratgozel%2Frobotstxt-util/lists"}