{"id":18537318,"url":"https://github.com/elixir-crawly/crawly_ui","last_synced_at":"2025-04-09T17:36:59.263Z","repository":{"id":42870406,"uuid":"258304916","full_name":"elixir-crawly/crawly_ui","owner":"elixir-crawly","description":"User interface for Crawly","archived":false,"fork":false,"pushed_at":"2023-01-06T04:18:09.000Z","size":4361,"stargazers_count":78,"open_issues_count":18,"forks_count":14,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-03-24T09:04:00.239Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/elixir-crawly.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":"oltarasenko"}},"created_at":"2020-04-23T19:08:50.000Z","updated_at":"2024-06-01T15:09:48.000Z","dependencies_parsed_at":"2023-02-05T07:02:12.856Z","dependency_job_id":null,"html_url":"https://github.com/elixir-crawly/crawly_ui","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elixir-crawly%2Fcrawly_ui","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elixir-crawly%2Fcrawly_ui/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elixir-crawly%2Fcrawly_ui/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elixir-crawly%2Fcrawly_ui/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/elixir-crawly","download_url":"https://codeload.github.com/elixir-crawly/crawly_ui/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248078271,"owners_count":21044080,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T19:37:48.714Z","updated_at":"2025-04-09T17:36:57.928Z","avatar_url":"https://github.com/elixir-crawly.png","language":"Elixir","funding_links":["https://github.com/sponsors/oltarasenko"],"categories":[],"sub_categories":[],"readme":"# CrawlyUI\n\n[![Build Status](https://travis-ci.org/oltarasenko/crawly_ui.svg?branch=master)](https://travis-ci.org/github/oltarasenko/crawly_ui)\n[![Coverage Status](https://coveralls.io/repos/github/oltarasenko/crawly_ui/badge.svg?branch=master)](https://coveralls.io/github/oltarasenko/crawly_ui?branch=master)\n\n## Motivation for the project\n\nWeb scraping is simple. You can easily fetch many pages with CURL or any other\nold good client. However we don't believe it :)!\n\nWe thing that web scraping is complex if you do it commercially.\nMost of the times we see that, indeed people can extract pages with simple curl\nbased client (or Floki + something), however they normally fail if it comes about\nclear deliverable.\n\nWe think that Crawling is hard when it comes to:\n1. Scalability - imagine you have SLA to deliver another million of items before the EOD\n2. Data quality - it's easy to get some pages, it's hard to extract the right data\n3. Javascript - most of the platforms would not even mention it :)\n4. Coverage - what if you need to get literally all pages/products, do you really\n   plan to check it all manually using the command line?\n\n## Our approach\n\nWe have created a platform which allows to manage jobs and to visualize items in the\nnice way. We believe it's important to see how your data looks! It's important to\nbe able to filter it, analyzing every job and comparing jobs.\n\nAnd finally it's important to be able to compare extracted data with the data on\nthe target website.\n\nWe think that web scraping is a process! It involves development, debugging, QA\nand finally maintenance. And that's what we're trying to achieve with CrawlyUI\nproject.\n\n## Trying it\n\n### Live demo\nLive demo is available at:\nhttp://crawlyui.com/\n\n### Locally\nYou could run it locally using the following commands\n\n1. `docker-compose build`\n2. `docker-compose up -d postgres`\n3. `docker-compose run ui bash -c \"/crawlyui/bin/ec eval \\\"CrawlyUI.ReleaseTasks.migrate\\\"\"`\n4. `docker-compose up ui worker`\n\nThis should bring the Crawly UI, Crawly worker (with Crawly Jobs in example folder)\nand postgres database for you. Now you can access the server from localhost:80\n\n## Gallery\n\n1. Main page. Schedule jobs here!\n![Main Page](gallery/main_page.png?raw=true)\n\n2. All jobs page\n![All Jobs](gallery/all_jobs_page.png?raw=true)\n\n3. Items browser\n![Items browser](gallery/items_page.png?raw=true)\n![Items browser search](gallery/item_with_filters.png?raw=true)\n\n4. Items preview\n![Items browser](gallery/item_preview_example.png?raw=true)\n\n## How it works\n\nCrawlyUI is a phoenix application, which is responsible for working with Crawly\nnodes. All nodes are connected to CrawlyUI using the erlang distribution. CrawlyUI\ncan operate as many nodes as you want (or as many as erlang distribution can handle ~100)\n\n# Testing it on localhost (with your own Crawly jobs)\n\n## Your own Crawly Implementation\n\n### Configure your Crawly\n\n1. Add SendToUI pipeline to the list of your item pipelines (before encoder pipelines)\n`{Crawly.Pipelines.Experimental.SendToUI, ui_node: :\"\u003ccrawlyui-node-name\u003e@127.0.0.1\"}`, example:\n\n``` elixir\n# config/config.exs\nconfig :crawly,\n  pipelines: [\n     {Crawly.Pipelines.Validate, fields: [:id]},\n     {Crawly.Pipelines.DuplicatesFilter, item_id: :id},\n     {Crawly.Pipelines.Experimental.SendToUI, ui_node: :\"ui@127.0.0.1\"},\n     Crawly.Pipelines.JSONEncoder,\n     {Crawly.Pipelines.WriteToFile, folder: \"/tmp\", extension: \"jl\"}\n  ]\n```\n\n2. Organize erlang cluster so Crawly nodes can find CrawlyUI node, in this case we use\n[erlang-node-discovery](https://github.com/oltarasenko/erlang-node-discovery) application\nfor this task, however any other alternative would also work. For setting up\nerlang-node-discovery:\n\n- add erlang-node-discovery as dependencies:\n\n``` elixir\n  # mix.exs\n  defp deps do\n    [\n      {:erlang_node_discovery, git: \"https://github.com/oltarasenko/erlang-node-discovery\"}\n    ]\n```\n\n- add the following to the configuration:\n\n``` elixir\n  # config/config.exs\n  config :erlang_node_discovery, hosts: [\"127.0.0.1\"], node_ports: [{\u003ccrawlyui-node-name\u003e, 4000}]\n```\n\nwhere `\u003ccrawlyui-node-name\u003e` needs to match with the `SendToUi` pipeline (it would be `{:ui, 4000}`\nto match with the example)\n\n### Start your Crawly node\nStart an iex session in your Crawly implementation directory with `--cookie`\n(which should be same with your CrawlyUI session), you can also define a node name\nwith option `--name` and it will be your Crawly node name that shows up on the UI, example:\n\n``` bash\n$ iex --name worker@worker.com --cookie 123 -S mix\n```\n\n## CrawlyUI\n\n### Start the database\n\nStart postgres with the command\n\n``` bash\n$ docker-compose build\n$ docker-compose up -d postgres\n```\n\n### Start CrawlyUI session\n\nStart an iex session in CrawlyUI directory with `--name \u003ccrawlyui-node-name\u003e@127.0.0.1`\nand `--cookie` that is the same with the crawly session, example:\n\n``` bash\n$ iex --name ui@127.0.0.1 --cookie 123 -S mix phx.server\n```\n\nThe interface will be available on [localhost:4000]() for your tests.\n\n## Note\nIf the crawly node does not show up on [http://localhost:4000/schedule](), try on your Crawly iex\nsession ping the CrawlyUI node with `Node.ping/1`, example:\n\n```bash\niex (worker@worker.com) 1\u003e Node.ping(:\"ui@127.0.0.1\")\n\u003e :pong\n```\n\nIf you get `:pong` back, refresh the page and the crawly node will be there\n\n# Item previews\n\nIf your item has URL field you will get a nice preview capabilities, with the\nhelp of iframe.\n\nNOTE:\n\nIframes are blocked almost by every large website. However you can easily overcome it by\nusing Ignore X-Frame headers browser extension.\n\n# Roadmap\n\n- [x] Make tests to have 80% tests coverage\n- [ ] Get logs from Crawly\n- [ ] Allow to stop given spiders\n- [ ] Parametrize spider parameters\n- [ ] Export items (CSV, JSON)\n- [ ] Make better search (search query language like in Kibana)\n- [ ] UI based spider generation\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felixir-crawly%2Fcrawly_ui","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Felixir-crawly%2Fcrawly_ui","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felixir-crawly%2Fcrawly_ui/lists"}