https://github.com/sultaniman/dust

Archive web pages with all relevant assets or save as a single file HTML
https://github.com/sultaniman/dust

elixir http http-requests scraping

Last synced: 7 days ago
JSON representation

Archive web pages with all relevant assets or save as a single file HTML

• Host: GitHub
• URL: https://github.com/sultaniman/dust
• Owner: sultaniman
• License: apache-2.0
• Created: 2020-05-16T19:38:46.000Z (over 4 years ago)
• Default Branch: develop
• Last Pushed: 2023-07-11T18:40:12.000Z (over 1 year ago)
• Last Synced: 2024-05-01T22:26:36.835Z (7 months ago)
• Topics: elixir, http, http-requests, scraping
• Language: Elixir
• Homepage:
• Size: 251 KB
• Stars: 19
• Watchers: 1
• Forks: 0
• Open Issues: 3
• Metadata Files:
  • Readme: README.md
  • Contributing: CONTRIBUTING.md
  • Funding: .github/FUNDING.yml
  • License: LICENSE
  • Code of conduct: CODE_OF_CONDUCT.md

Awesome Lists containing this project

README

        
Dust



  



**NOTE:** Please note this project is still under development so you might experience issues.

## Installation 💾

If [available in Hex](https://hex.pm/docs/publish), the package can be installed

by adding `dust` to your list of dependencies in `mix.exs`:

```elixir

def deps do

  [

    {:dust, "~> 0.0.2-dev"}

  ]

end

```

## Usage 🧠

```elixir

"https://github.com"

|> Dust.get()

|> Dust.persist("AWESOME/PAGE.HTML")

"https://times.com"

|> Dust.get(

  headers: headers,

  proxy: %Proxy{...} | "socks5://user:[email protected]:port",

  max_retries: 3

)

|> Dust.persist(result, "AWESOME/PAGE.HTML")

```

## Documentation

Documentation can be generated with [ExDoc](https://github.com/elixir-lang/ex_doc)

and published on [HexDocs](https://hexdocs.pm). Once published, the docs can

be found at [https://hexdocs.pm/dust](https://hexdocs.pm/dust).

## Assets 💄

https://www.flaticon.com/free-icon/dust_867847