{"id":19994199,"url":"https://github.com/b4dnewz/robots-parse","last_synced_at":"2025-09-22T03:05:18.713Z","repository":{"id":25538548,"uuid":"104403600","full_name":"b4dnewz/robots-parse","owner":"b4dnewz","description":"A lightweight and simple robots.txt parser in node","archived":false,"fork":false,"pushed_at":"2023-02-27T16:30:52.000Z","size":663,"stargazers_count":7,"open_issues_count":17,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-08-09T15:33:12.533Z","etag":null,"topics":["osint","parser","robots-parser","robots-txt"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/b4dnewz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-09-21T22:08:07.000Z","updated_at":"2022-02-17T02:34:35.000Z","dependencies_parsed_at":"2025-05-04T13:36:27.690Z","dependency_job_id":"51f40271-7c14-47de-a707-43b528d7a851","html_url":"https://github.com/b4dnewz/robots-parse","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/b4dnewz/robots-parse","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/b4dnewz%2Frobots-parse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/b4dnewz%2Frobots-parse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/b4dnewz%2Frobots-parse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/b4dnewz%2Frobots-parse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/b4dnewz","download_url":"https://codeload.github.com/b4dnewz/robots-parse/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/b4dnewz%2Frobots-parse/sbom","scorecard":{"id":222140,"data":{"date":"2025-08-11","repo":{"name":"github.com/b4dnewz/robots-parse","commit":"63fa1c2dcd459204c6d54d1a74ac134311a37d41"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":1.7,"checks":[{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":0,"reason":"65 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GHSA-968p-4wvh-cqc8","Warn: Project is vulnerable to: GHSA-67hx-6x53-jw92","Warn: Project is vulnerable to: GHSA-6chw-6frg-f759","Warn: Project is vulnerable to: GHSA-v88g-cgmw-v5xw","Warn: Project is vulnerable to: GHSA-93q8-gq69-wqmw","Warn: Project is vulnerable to: GHSA-qwcr-r2fm-qrc7","Warn: Project is vulnerable to: GHSA-v6h2-p8h4-qcjw","Warn: Project is vulnerable to: GHSA-grv7-fg5c-xmjg","Warn: Project is vulnerable to: GHSA-pxg6-pf52-xh8x","Warn: Project is vulnerable to: GHSA-3xgq-45jj-v275","Warn: Project is vulnerable to: GHSA-gxpj-cx7g-858c","Warn: Project is vulnerable to: GHSA-w573-4hg7-7wgq","Warn: Project is vulnerable to: GHSA-rv95-896h-c2vc","Warn: Project is vulnerable to: GHSA-qw6h-vgh9-j6wx","Warn: Project is vulnerable to: GHSA-74fj-2j2h-c42q","Warn: Project is vulnerable to: GHSA-pw2r-vq6v-hr8c","Warn: Project is vulnerable to: GHSA-jchw-25xp-jwwc","Warn: Project is vulnerable to: GHSA-cxjh-pqwp-8mfp","Warn: Project is vulnerable to: GHSA-fjxv-7rqg-78g4","Warn: Project is vulnerable to: GHSA-8r6j-v8pm-fqw3","Warn: Project is vulnerable to: MAL-2023-462","Warn: Project is vulnerable to: GHSA-4q6p-r6v2-jvc5","Warn: Project is vulnerable to: GHSA-w457-6q6x-cgp9","Warn: Project is vulnerable to: GHSA-62gr-4qp9-h98f","Warn: Project is vulnerable to: GHSA-f52g-6jhx-586p","Warn: Project is vulnerable to: GHSA-2cf5-4w76-r9qv","Warn: Project is vulnerable to: GHSA-3cqr-58rm-57f8","Warn: Project is vulnerable to: GHSA-g9r4-xpmj-mj65","Warn: Project is vulnerable to: GHSA-q2c6-c6pm-g3gh","Warn: Project is vulnerable to: GHSA-765h-qjxv-5f44","Warn: Project is vulnerable to: GHSA-f2jv-r9rf-7988","Warn: Project is vulnerable to: GHSA-43f8-2h32-f4cj","Warn: Project is vulnerable to: GHSA-qqgx-2p2h-9c37","Warn: Project is vulnerable to: GHSA-896r-f27r-55mw","Warn: Project is vulnerable to: GHSA-9c47-m6qq-7p4h","Warn: Project is vulnerable to: GHSA-6c8f-qphg-qjgp","Warn: Project is vulnerable to: GHSA-p6mc-m468-83gw","Warn: Project is vulnerable to: GHSA-29mw-wpgm-hmr9","Warn: Project is vulnerable to: GHSA-35jh-r3h4-6jhm","Warn: Project is vulnerable to: GHSA-952p-6rrq-rcjv","Warn: Project is vulnerable to: GHSA-f8q6-p94x-37v3","Warn: Project is vulnerable to: GHSA-vh95-rmgr-6w4m","Warn: Project is vulnerable to: GHSA-xvch-5gv4-984h","Warn: Project is vulnerable to: GHSA-5fw9-fq32-wv5p","Warn: Project is vulnerable to: GHSA-hj48-42vr-x3v9","Warn: Project is vulnerable to: GHSA-9wv6-86v2-598j","Warn: Project is vulnerable to: GHSA-rhx6-c78j-4q9w","Warn: Project is vulnerable to: GHSA-g6ww-v8xp-vmwg","Warn: Project is vulnerable to: GHSA-hrpp-h998-j3pp","Warn: Project is vulnerable to: GHSA-p8p7-x288-28g6","Warn: Project is vulnerable to: GHSA-c2qf-rxjj-qqgw","Warn: Project is vulnerable to: GHSA-m6fv-jmcg-4jfg","Warn: Project is vulnerable to: GHSA-cm22-4g7w-348p","Warn: Project is vulnerable to: GHSA-3jfq-g458-7qm9","Warn: Project is vulnerable to: GHSA-r628-mhmh-qjhw","Warn: Project is vulnerable to: GHSA-9r2w-394v-53qc","Warn: Project is vulnerable to: GHSA-5955-9wpr-37jh","Warn: Project is vulnerable to: GHSA-qq89-hq3f-393p","Warn: Project is vulnerable to: GHSA-f5x3-32g6-xq36","Warn: Project is vulnerable to: GHSA-jgrx-mgxx-jf9v","Warn: Project is vulnerable to: GHSA-72xf-g2v4-qvf3","Warn: Project is vulnerable to: GHSA-6fc8-4gx4-v693","Warn: Project is vulnerable to: GHSA-3h5v-q93c-6h6q","Warn: Project is vulnerable to: GHSA-c4w7-xm78-47vh","Warn: Project is vulnerable to: GHSA-p9pc-299p-vxgp"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-17T02:49:11.646Z","repository_id":25538548,"created_at":"2025-08-17T02:49:11.647Z","updated_at":"2025-08-17T02:49:11.647Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271400259,"owners_count":24752830,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-20T02:00:09.606Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["osint","parser","robots-parser","robots-txt"],"created_at":"2024-11-13T04:54:18.948Z","updated_at":"2025-09-22T03:05:13.667Z","avatar_url":"https://github.com/b4dnewz.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# robots-parse\n\n[![NPM version][npm-image]][npm-url] [![Build Status][travis-image]][travis-url] [![Dependency Status][daviddm-image]][daviddm-url] [![Coverage percentage][coveralls-image]][coveralls-url]\n\n\u003e A lightweight and simple robots.txt parser in node.\n\n[![NPM](https://nodei.co/npm/robots-parse.png)](https://nodei.co/npm/robots-parse/)\n\n## Installation\n\n```\nnpm install robots-parse\n```\n\n## Usage\n\nYou can use the module to scan a **domain** for robots file like in the example below:\n\n```js\nconst robotsParse = require('robots-parse');\n\nrobotsParse('github.com', (err, res) =\u003e {\n  console.log('Result:', res);\n});\n```\n\nYou can also use it with __promises__ if the callback is not specified:\n\n```js\nimport robotsParse from 'robots-parse'\n\n(async () =\u003e {\n  const res = await robotsParse('github.com');\n  console.log('Result:', res);\n})().catch(console.error)\n```\n\nOr you can use the built-in parser to parse an existing robots.txt file, for example a **local file** or a **string**. The parser works __in sync__ so you don't have to use callback or promises.\n\n```js\nconst {parser} = require('robots-parse');\n\nrequest('google.com/robots.txt', (err, res, body) =\u003e {\n  const object = parser(body);\n  console.log(object);\n});\n```\n\nParsing an existing local robots.txt file:\n\n```js\nimport {parser} from 'robots-parse'\n\nconst content = fs.readFileSync('./robots.txt', 'utf-8');\nconst object = parser(content);\n\nconsole.log(object);\n```\n\n## How it works?\n\nBy default the script will get and parse the `robots.txt` file for a given website or domain and it will search for various rules:\n\n-   **Agents**: A user-agent identifies a specific spider. The user-agent field is matched against that specific spider’s (usually longer) user-agent.\n-   **Host**: Supported by Yandex (and not by Google even though some posts say it does), this directive lets you decide whether you want the search engine to show.\n-   **Allow**: The allow directive specifies paths that may be accessed by the designated crawlers. When no path is specified, the directive is ignored.\n-   **Disallow**: The disallow directive specifies paths that must not be accessed by the designated crawlers. When no path is specified, the directive is ignored.\n-   **Sitemap**: An absolute url that points to a Sitemap, Sitemap Index file or equivalent URL.\n\nIt returns, _if the robots files were successfully retrieved and parsed_, an object containing the properties mentioned above, inside every agent found you will find agent-specific **allow** and **disallow** rules, which also will be stored in **allow** and **disallow**  root properties containing all of them indistinctly.\n\nYou can read more about the specifications of the robots file on it's [Google Reference Page](https://developers.google.com/search/reference/robots_txt).\n\n---\n\n## Contributing\n\n1.  Create an issue and describe your idea\n2.  Fork the project (\u003chttps://github.com/b4dnewz/robots-parse/fork\u003e)\n3.  Create your feature branch (`git checkout -b my-new-feature`)\n4.  Commit your changes (`git commit -am 'Add some feature'`)\n5.  Write tests for your code (`npm run test`)\n6.  Publish the branch (`git push origin my-new-feature`)\n7.  Create a new Pull Request\n\n## License\n\nMIT © [b4dnewz](https://b4dnewz.github.io/)\n\n[npm-image]: https://badge.fury.io/js/robots-parse.svg\n\n[npm-url]: https://npmjs.org/package/robots-parse\n\n[travis-image]: https://travis-ci.org/b4dnewz/robots-parse.svg?branch=master\n\n[travis-url]: https://travis-ci.org/b4dnewz/robots-parse\n\n[daviddm-image]: https://david-dm.org/b4dnewz/robots-parse.svg?theme=shields.io\n\n[daviddm-url]: https://david-dm.org/b4dnewz/robots-parse\n\n[coveralls-image]: https://coveralls.io/repos/b4dnewz/robots-parse/badge.svg\n\n[coveralls-url]: https://coveralls.io/r/b4dnewz/robots-parse\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fb4dnewz%2Frobots-parse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fb4dnewz%2Frobots-parse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fb4dnewz%2Frobots-parse/lists"}