{"id":31702320,"url":"https://github.com/ravern/gollum","last_synced_at":"2025-12-11T23:32:39.175Z","repository":{"id":44430966,"uuid":"106017826","full_name":"ravern/gollum","owner":"ravern","description":"Robots.txt parser and fetcher for Elixir","archived":false,"fork":false,"pushed_at":"2023-03-30T22:08:47.000Z","size":30,"stargazers_count":14,"open_issues_count":3,"forks_count":12,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-07T00:47:50.689Z","etag":null,"topics":["crawler","elixir","robots-parser","robots-txt"],"latest_commit_sha":null,"homepage":"https://hexdocs.pm/gollum","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ravern.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2017-10-06T15:03:21.000Z","updated_at":"2024-05-23T11:28:16.000Z","dependencies_parsed_at":"2023-07-12T16:46:38.934Z","dependency_job_id":null,"html_url":"https://github.com/ravern/gollum","commit_stats":null,"previous_names":["ravernkoh/gollum"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/ravern/gollum","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ravern%2Fgollum","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ravern%2Fgollum/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ravern%2Fgollum/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ravern%2Fgollum/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ravern","download_url":"https://codeload.github.com/ravern/gollum/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ravern%2Fgollum/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279000701,"owners_count":26082837,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","elixir","robots-parser","robots-txt"],"created_at":"2025-10-08T21:50:51.476Z","updated_at":"2025-10-08T21:50:52.789Z","avatar_url":"https://github.com/ravern.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://semaphoreci.com/api/v1/ravernkoh/gollum/branches/master/shields_badge.svg)](https://semaphoreci.com/ravernkoh/gollum)\n\n# Gollum\nRobots.txt parser with caching. Modelled after Kryten. Docs can be found [here](https://hexdocs.pm/gollum/api-reference.html).\n\n# Usage\nCall Gollum.crawlable?/3 to obtain whether a certain URL is permitted for the specified user agent.\n```elixir\niex\u003e Gollum.crawlable?(\"hello\", \"https://google.com/\")\n:crawlable\niex\u003e Gollum.crawlable?(\"hello\", \"https://google.com/m/\")\n:uncrawlable\n```\n\nGollum is an OTP app (For the cache) so just remember to specify it in the extra_applications key in your mix.exs to ensure it is started.\n\nGollum allows for some configuration in your config.exs file. The following shows their default values. They are all optional.\n```elixir\nconfig :gollum,\n  name: Gollum.Cache, # Name of the Cache GenServer\n  refresh_secs: 86_400, # Amount of time before the robots.txt will be refetched\n  lazy_refresh: false, # Whether to setup a timer that auto-refetches, or to only refetch when requested\n  user_agent: \"Gollum\" # User agent to use when sending the GET request for the robots.txt\n```\n\n# Author\nRavern Koh - \u003c\u003cravern.koh.dev@gmail.com\u003e\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fravern%2Fgollum","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fravern%2Fgollum","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fravern%2Fgollum/lists"}