https://github.com/wabarc/rivet
A toolkit makes it easier to archive webpages to IPFS
https://github.com/wabarc/rivet
archiving ipfs webpage
Last synced: 5 months ago
JSON representation
A toolkit makes it easier to archive webpages to IPFS
- Host: GitHub
- URL: https://github.com/wabarc/rivet
- Owner: wabarc
- License: mit
- Created: 2021-12-28T13:57:10.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2023-07-31T19:48:50.000Z (about 2 years ago)
- Last Synced: 2025-05-10T22:02:48.055Z (5 months ago)
- Topics: archiving, ipfs, webpage
- Language: Go
- Homepage:
- Size: 151 KB
- Stars: 12
- Watchers: 2
- Forks: 1
- Open Issues: 8
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Rivet
[](https://github.com/wabarc/rivet/blob/main/LICENSE)
[](https://goreportcard.com/report/github.com/wabarc/rivet)
[](https://github.com/wabarc/rivet/actions/workflows/builder.yml)
[](https://pkg.go.dev/github.com/wabarc/rivet)
[](https://github.com/wabarc/rivet/releases)
Rivet is both a command-line tool and a Golang package for archiving webpages to IPFS.
Supported Golang version: See [.github/workflows/testing.yml](./.github/workflows/testing.yml)
## Installation
The simplest, cross-platform way is to download from [GitHub Releases](https://github.com/wabarc/rivet/releases) and place the executable file in your PATH.
From source:
```sh
go get -u github.com/wabarc/rivet/cmd/rivet
```
From GitHub Releases:
```sh
sh <(wget https://github.com/wabarc/rivet/raw/main/install.sh -O-)
```
Using [Snapcraft](https://snapcraft.io/rivet) (on GNU/Linux)
```sh
sudo snap install rivet
```
## Usage
### Command line
```sh
A toolkit makes it easier to archive webpages to IPFS.
Usage:
rivet [options] [url1] ... [urlN]
-host string
IPFS node address (default "localhost")
-m string
Pin mode, supports mode: local, remote, archive (default "remote")
-p string
Pinner sceret or password.
-port int
IPFS node port (default 5001)
-t string
IPFS pinner, supports pinners: infura, pinata, nftstorage, web3storage. (default "infura")
-timeout uint
Timeout for every input URL (default 30)
-u string
Pinner apikey or username.
```
#### Examples
Stores data on local IPFS node.
```sh
rivet -m local https://example.com https://example.org
```
Stores data to remote pinning services.
```sh
rivet https://example.com
```
Or, specify a pinning service.
```sh
rivet -t pinata -k your-apikey -s your-secret https://example.com
```
Or, stores file locally without any IPFS node.
```sh
rivet -m archive https://example.com
```
### Go package
```go
package main
import (
"context"
"fmt"
"net/url"
"github.com/wabarc/ipfs-pinner"
"github.com/wabarc/rivet"
"github.com/wabarc/rivet/ipfs"
)
func main() {
opts := []ipfs.PinningOption{
ipfs.Mode(ipfs.Remote),
ipfs.Uses(pinner.Infura),
}
p := ipfs.Options(opts...)
r := &rivet.Shaft{Hold: p}
l := "https://example.com"
input, err := url.Parse(l)
if err != nil {
panic(err)
}
dst, err := r.Wayback(context.TODO(), input)
if err != nil {
panic(err)
}
fmt.Println(dst)
}
```
## F.A.Q
### Optional to disable JavaScript for some URI?
If you prefer to disable JavaScript on saving webpage, you could add environmental variables `DISABLEJS_URIS`
and set the values with the following formats:
```sh
export DISABLEJS_URIS=wikipedia.org|eff.org/tags
```
It will disable JavaScript for domain of the `wikipedia.org` and path of the `eff.org/tags` if matching it.
## Credit
Special thanks to [@RadhiFadlillah](https://github.com/RadhiFadlillah) for making [obelisk](https://github.com/go-shiori/obelisk), under which the crawling of the web is based.
## Contributing
We encourage all contributions to this repository! Open an issue! Or open a Pull Request!
## License
This software is released under the terms of the MIT. See the [LICENSE](https://github.com/wabarc/rivet/blob/main/LICENSE) file for details.