{"id":13563487,"url":"https://github.com/feng19/spider_man","last_synced_at":"2025-04-03T02:11:43.518Z","repository":{"id":48009131,"uuid":"353759678","full_name":"feng19/spider_man","owner":"feng19","description":"SpiderMan,a base-on Broadway fast high-level web crawling \u0026 scraping framework for Elixir.","archived":false,"fork":false,"pushed_at":"2024-12-24T02:05:52.000Z","size":216,"stargazers_count":25,"open_issues_count":0,"forks_count":4,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-24T08:41:21.994Z","etag":null,"topics":["crawler","data-mining","elixir","erlang","framework","spider"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/feng19.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-01T16:22:13.000Z","updated_at":"2025-01-29T22:18:59.000Z","dependencies_parsed_at":"2023-01-29T20:30:55.512Z","dependency_job_id":"8e8f3676-a46a-43c0-b629-d6b325376ec9","html_url":"https://github.com/feng19/spider_man","commit_stats":{"total_commits":93,"total_committers":4,"mean_commits":23.25,"dds":"0.12903225806451613","last_synced_commit":"3557a896fdbdb7eb5c62f6feb99673f61a718d47"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feng19%2Fspider_man","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feng19%2Fspider_man/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feng19%2Fspider_man/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feng19%2Fspider_man/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/feng19","download_url":"https://codeload.github.com/feng19/spider_man/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246922248,"owners_count":20855345,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","data-mining","elixir","erlang","framework","spider"],"created_at":"2024-08-01T13:01:19.816Z","updated_at":"2025-04-03T02:11:43.499Z","avatar_url":"https://github.com/feng19.png","language":"Elixir","funding_links":[],"categories":["HTTP"],"sub_categories":[],"readme":"# SpiderMan\n\n[![Module Version](https://img.shields.io/hexpm/v/spider_man.svg)](https://hex.pm/packages/spider_man)\n[![Hex Docs](https://img.shields.io/badge/hex-docs-lightgreen.svg)](https://hexdocs.pm/spider_man/)\n[![Total Download](https://img.shields.io/hexpm/dt/spider_man.svg)](https://hex.pm/packages/spider_man)\n[![License](https://img.shields.io/hexpm/l/spider_man.svg)](https://github.com/feng19/spider_man/blob/master/LICENSE)\n[![Last Updated](https://img.shields.io/github/last-commit/feng19/spider_man.svg)](https://github.com/feng19/spider_man/commits/master)\n\n**SpiderMan,a fast high-level web crawling \u0026 scraping framework for Elixir.**\n\nInspired by [Crawly](https://github.com/elixir-crawly/crawly)(Elixir) and [Scrapy](https://github.com/scrapy/scrapy)(Python).\n\n## Usage\n\nThis example show how to crawly data from\n[elixir-jobs](https://elixirjobs.net/) website by use SpiderMan.\n\n### Use in livebook\n\n[![Run in Livebook](https://livebook.dev/badge/v1/blue.svg)](https://livebook.dev/run?url=https%3A%2F%2Fhexdocs.pm%2Fspider_man%2Felixirjobs.livemd)\n\n### Use in script or mix project\n\n```elixir\nMix.install([\n  {:spider_man, \"~\u003e 0.6\"},\n  {:floki, \"~\u003e 0.34\"},\n  {:nimble_csv, \"~\u003e 1.2\"}\n])\n\ndefmodule SpiderList.ElixirJobs do\n  @moduledoc false\n  use SpiderMan\n  require Logger\n  alias SpiderMan.Response\n\n  @base_url \"https://elixirjobs.net/\"\n\n  def run do\n    SpiderMan.run_until_zero(__MODULE__, [], 3_000)\n  end\n\n  @impl true\n  def settings do\n    requester_options = [\n      base_url: @base_url,\n      middlewares: [\n        {SpiderMan.Middleware.UserAgent,\n         [\n           \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36\",\n           \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36\",\n           \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36\",\n           \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4389.82 Safari/537.36\",\n           \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4389.82 Safari/537.36\",\n           \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36\"\n         ]},\n        {Tesla.Middleware.Headers,\n         [\n           {\"referer\", @base_url},\n           {\"accept\",\n            \"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"},\n           {\"accept-encoding\", \"gzip, deflate\"},\n           {\"accept-language\", \"zh-CN,zh;q=0.9,zh-TW;q=0.8,en;q=0.7\"}\n         ]},\n        Tesla.Middleware.DecompressResponse\n      ]\n    ]\n\n    [\n      downloader_options: [requester: {SpiderMan.Requester.Finch, requester_options}],\n      spider_options: [pipelines: []],\n      item_processor_options: [\n        storage: [\n          {SpiderMan.Storage.ETS, \"./data/jobs.ets\"},\n          {SpiderMan.Storage.CSV,\n           file: \"./data/jobs.csv\",\n           headers: [\n             :link,\n             :title,\n             :sub_title,\n             :date,\n             :workplace,\n             :type\n           ]}\n        ]\n      ]\n    ]\n  end\n\n  @impl true\n  def init(state) do\n    build_request(@base_url)\n    |\u003e set_flag(:first_page)\n    |\u003e then(\u0026SpiderMan.insert_request(__MODULE__, \u00261))\n\n    state\n  end\n\n  @impl true\n  def handle_response(%Response{env: env, flag: :first_page}, _context) do\n    total_page =\n      Regex.run(~r/Showing page 1 of (\\d+)/, env.body, capture: :all_but_first)\n      |\u003e hd()\n      |\u003e String.to_integer()\n\n    Logger.info(\"total: #{total_page}\")\n\n    requests =\n      Enum.map(2..total_page, fn n -\u003e\n        build_request(\"/?page=#{n}\")\n        |\u003e set_flag({:list_page, n})\n      end)\n\n    handle_list_page(env.body, 1)\n    |\u003e Map.put(:requests, requests)\n  end\n\n  def handle_response(%Response{env: env, flag: {:list_page, n}}, _context) do\n    handle_list_page(env.body, n)\n  end\n\n  defp handle_list_page(body, n) do\n    Logger.info(\"processing page #{n}\")\n    {:ok, document} = Floki.parse_document(body)\n\n    jobs =\n      Floki.find(document, \".offers-index\")\n      |\u003e hd()\n      |\u003e Floki.children(include_text: false)\n      |\u003e Enum.filter(\u0026match?({\"a\", _, _}, \u00261))\n\n    items =\n      Enum.map(jobs, fn job -\u003e\n        title = Floki.find(job, \".title strong\") |\u003e Floki.text() |\u003e String.trim()\n        sub_title = Floki.find(job, \".title small\") |\u003e Floki.text() |\u003e String.trim()\n        link = Floki.attribute(job, \"a\", \"href\") |\u003e hd()\n\n        [_, date, _, workplace, _, type] =\n          Floki.find(job, \".control .tag\")\n          |\u003e Enum.map(\u0026(\u00261 |\u003e Floki.text() |\u003e String.trim()))\n\n        build_item(\n          link,\n          %{\n            link: @base_url \u003c\u003e String.slice(link, 1..-1),\n            title: title,\n            sub_title: sub_title,\n            date: date,\n            workplace: workplace,\n            type: type\n          }\n        )\n      end)\n\n    %{items: items}\n  end\nend\n\nSpiderList.ElixirJobs.run()\n```\n\ncopy this script and save to `elixir_jobs.exs` and then start by command:\n```shell\nelixir elixir_jobs.exs\n```\n\n## Copyright and License\n\nCopyright (c) 2023 feng19\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at [http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffeng19%2Fspider_man","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffeng19%2Fspider_man","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffeng19%2Fspider_man/lists"}