{"id":17678545,"url":"https://github.com/antoinegagne/robots","last_synced_at":"2026-02-27T10:05:16.922Z","repository":{"id":48952119,"uuid":"224552280","full_name":"AntoineGagne/robots","owner":"AntoineGagne","description":"A parser for robots.txt with support for wildcards. See also RFC 9309.","archived":false,"fork":false,"pushed_at":"2023-11-21T21:07:56.000Z","size":31,"stargazers_count":3,"open_issues_count":3,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-01T04:07:24.986Z","etag":null,"topics":["crawling","erlang","erlang-library","parser","parsing","parsing-library","rfc-9309","robots-exclusion-standard","robots-parser","robots-txt"],"latest_commit_sha":null,"homepage":"","language":"Erlang","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AntoineGagne.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-11-28T02:14:09.000Z","updated_at":"2024-12-11T20:38:46.000Z","dependencies_parsed_at":"2023-11-21T21:49:19.298Z","dependency_job_id":null,"html_url":"https://github.com/AntoineGagne/robots","commit_stats":{"total_commits":15,"total_committers":3,"mean_commits":5.0,"dds":0.4,"last_synced_commit":"6405fdb336e40789f0a00a90a914234e4911591d"},"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AntoineGagne%2Frobots","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AntoineGagne%2Frobots/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AntoineGagne%2Frobots/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AntoineGagne%2Frobots/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AntoineGagne","download_url":"https://codeload.github.com/AntoineGagne/robots/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249942341,"owners_count":21349002,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawling","erlang","erlang-library","parser","parsing","parsing-library","rfc-9309","robots-exclusion-standard","robots-parser","robots-txt"],"created_at":"2024-10-24T08:05:10.199Z","updated_at":"2026-02-27T10:05:15.928Z","avatar_url":"https://github.com/AntoineGagne.png","language":"Erlang","funding_links":[],"categories":[],"sub_categories":[],"readme":"# robots\n\n[![Build Status](https://github.com/AntoineGagne/robots/actions/workflows/erlang.yml/badge.svg)](https://github.com/AntoineGagne/robots/actions)\n[![Hex Pm](http://img.shields.io/hexpm/v/robots.svg?style=flat)](https://hex.pm/packages/robots)\n[![Docs](https://img.shields.io/badge/hex-docs-green.svg?style=flat)](https://hexdocs.pm/robots)\n[![Releases](https://img.shields.io/github/release/AntoineGagne/robots?color=brightgreen)](https://github.com/AntoineGagne/robots/releases)\n[![Coverage](https://coveralls.io/repos/github/AntoineGagne/robots/badge.svg?branch=master)](https://coveralls.io/github/AntoineGagne/robots?branch=master)\n\nA library that parses and validates rules from `robots.txt`.\n\n## Installation\n\nThis library is available on [hex.pm](https://hex.pm/packages/robots).\nTo install this library, simply add the following lines to your\n`rebar.config`:\n\n```erlang\n{robots, \"1.1.1\"}\n```\n\n## Usage\n\n```erlang\nContent = \u003c\u003c\"User-Agent: bot\\nAllow: /fish\"\u003e\u003e,\n%% This will return an opaque type that contains all the rules and their agents\n{ok, RulesIndex} = robots:parse(Content, 200),\ntrue = robots:is_allowed(\u003c\u003c\"bot/1.0.0\"\u003e\u003e, \u003c\u003c\"/fish/salmon.html\"\u003e\u003e, RulesIndex),\ntrue = robots:is_allowed(\u003c\u003c\"bot/1.0.0\"\u003e\u003e, \u003c\u003c\"/Fish.asp\"\u003e\u003e, RulesIndex),\n```\n\n## Development\n\n### Running all the tests and linters\n\nYou can run all the tests and linters with the `rebar3` alias:\n\n```sh\nrebar3 check\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fantoinegagne%2Frobots","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fantoinegagne%2Frobots","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fantoinegagne%2Frobots/lists"}