https://github.com/kytiken/euros
Euros web-spider framework
https://github.com/kytiken/euros
elixir
Last synced: 10 months ago
JSON representation
Euros web-spider framework
- Host: GitHub
- URL: https://github.com/kytiken/euros
- Owner: kytiken
- License: mit
- Created: 2017-12-12T07:41:42.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2019-03-10T09:31:25.000Z (over 7 years ago)
- Last Synced: 2025-04-02T17:02:47.644Z (about 1 year ago)
- Topics: elixir
- Language: Elixir
- Homepage:
- Size: 54.7 KB
- Stars: 2
- Watchers: 1
- Forks: 1
- Open Issues: 13
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Euros
Euros is a web spider framework that can spider a domain and collect useful information about the pages it visits.
It is versatile, allowing you to write your own specialized spider tasks quickly and easily.
## Installation
```elixir
def deps do
[
{:euros, "~> 0.3.0"}
]
end
```
## Usage
```elixir
iex> url = "https://euros-test.blogspot.jp/"
iex> Euros.Core.crawl(url, fn(page) -> IO.puts(inspect(page)) end)
%HTTPoison.Response{body: "euros test blog....
```
### set depth limit
```elixir
iex> url = "https://euros-test.blogspot.jp/"
iex> option = %Euros.CrawlOption{depth_limit: 2}
iex> Euros.Core.crawl(url, fn(page) -> page |> inspect |> IO.puts end, option)
```
### set cookie
```elixir
iex> url = "https://euros-test.blogspot.jp/"
iex> http_option = %Euros.HTTPOption{cookie: "foo=bar;"}
iex> option = %Euros.CrawlOption{http_option: http_option}
iex> Euros.Core.crawl(url, fn(page) -> page |> inspect |> IO.puts end, option)
```
### cusotm url pattern
```elixir
iex> url = "https://euros-test.blogspot.jp/"
iex> option = %Euros.HTTPOption{cookie: "foo=bar;"}
iex> pattern = ~r/test1/
iex> option = %Euros.CrawlOption{http_option: http_option, pattern: pattern}
iex> Euros.Core.crawl(url, fn(page) -> page |> inspect |> IO.puts end, option, pattern)
```
## License
```
Copyright (c) 2017 kytiken
Released under the MIT license
https://github.com/kytiken/euros/blob/master/LICENSE
```
Documentation can be generated with [ExDoc](https://github.com/elixir-lang/ex_doc)
and published on [HexDocs](https://hexdocs.pm). Once published, the docs can
be found at [https://hexdocs.pm/euros](https://hexdocs.pm/euros).