{"id":16702433,"url":"https://github.com/nietaki/crawlie","last_synced_at":"2025-08-24T13:18:56.688Z","repository":{"id":46342195,"uuid":"76183541","full_name":"nietaki/crawlie","owner":"nietaki","description":"A simple Elixir library for writing decently-performing crawlers with minimum effort.","archived":false,"fork":false,"pushed_at":"2021-10-30T00:04:43.000Z","size":201,"stargazers_count":90,"open_issues_count":8,"forks_count":11,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-07-02T12:54:35.101Z","etag":null,"topics":["crawler","elixir","elixir-library","genstage"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nietaki.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-12-11T15:42:59.000Z","updated_at":"2025-03-30T19:39:02.000Z","dependencies_parsed_at":"2022-09-01T22:31:49.774Z","dependency_job_id":null,"html_url":"https://github.com/nietaki/crawlie","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/nietaki/crawlie","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nietaki%2Fcrawlie","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nietaki%2Fcrawlie/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nietaki%2Fcrawlie/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nietaki%2Fcrawlie/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nietaki","download_url":"https://codeload.github.com/nietaki/crawlie/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nietaki%2Fcrawlie/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263385390,"owners_count":23458729,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","elixir","elixir-library","genstage"],"created_at":"2024-10-12T19:04:45.772Z","updated_at":"2025-07-03T19:06:49.903Z","avatar_url":"https://github.com/nietaki.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Crawlie (the crawler) [![badge](https://travis-ci.org/nietaki/crawlie.svg?branch=master)](https://travis-ci.org/nietaki/crawlie) [![Coverage Status](https://coveralls.io/repos/github/nietaki/crawlie/badge.svg?branch=master)](https://coveralls.io/github/nietaki/crawlie?branch=master) [![Hex.pm](https://img.shields.io/hexpm/v/crawlie.svg)](https://hex.pm/packages/crawlie) [![docs](https://img.shields.io/badge/docs-hexdocs-yellow.svg)](https://hexdocs.pm/crawlie/) [![Built with Spacemacs](https://cdn.rawgit.com/syl20bnr/spacemacs/442d025779da2f62fc86c2082703697714db6514/assets/spacemacs-badge.svg)](http://spacemacs.org)\n\n\nCrawlie is a simple Elixir library for writing decently-performing crawlers with minimum effort.\n\n## Usage example\n\nSee the [crawlie_example](https://github.com/nietaki/crawlie_example) project.\n\n## Inner workings\n\nCrawlie uses Elixir's [GenStage](https://github.com/elixir-lang/gen_stage) to parallelise\nthe work. Most of the logic is handled by the `Crawlie.Stage.UrlManager`, which consumes the url collection passed by the user, receives the urls extracted by the subsequent processing, makes sure no url is processed more than once, makes sure that the \"discovered urls\" collection is as small as possible by traversing the url tree in a roughly depth-first manner.\n\nThe urls are requested from the `Crawlie.Stage.UrlManager` by a GenStage [Flow](https://hexdocs.pm/flow/Flow.html#content), which in parallel\nfetches the urls using HTTPoison, and parses the responses using user-provided callbacks. Discovered urls get sent back to UrlManager.\n\nHere's a rough diagram:\n\n![crawlie architecture diagram](assets/crawlie_arch_v0.2.0.png)\n\n## Statistics\n\nIf you're interested in the crawling statistics or want to track the progress in real time, see [`Crawlie.crawl_and_track_stats/3`](https://hexdocs.pm/crawlie/Crawlie.html#crawl_and_track_stats/3). It starts a [`Stats GenServer`](https:/hexdocs.pm/crawlie/Crawlie.Stats.Server.html) in Crawlie's supervision tree, which accumulates the statistics for the crawling session.\n\n## Configuration\n\nSee [the docs](https://hexdocs.pm/crawlie/Crawlie.html#crawl/3) for supported options.\n\n## Installation\n\nThe package can be installed as:\n\n  1. Add `crawlie` to your list of dependencies in `mix.exs`:\n\n```elixir\ndef deps do\n  [{:crawlie, \"~\u003e 1.0.0\"}]\nend\n```\n\n  2. Ensure `crawlie` is started before your application:\n\n```elixir\ndef application do\n  [applications: [:crawlie]]\nend\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnietaki%2Fcrawlie","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnietaki%2Fcrawlie","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnietaki%2Fcrawlie/lists"}