Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/geut/permanent-seeder
A super seeder for hyperdrives. By GEUT.
https://github.com/geut/permanent-seeder
geut hyperdrive seeder vault
Last synced: 3 months ago
JSON representation
A super seeder for hyperdrives. By GEUT.
- Host: GitHub
- URL: https://github.com/geut/permanent-seeder
- Owner: geut
- License: mit
- Created: 2020-06-16T22:00:14.000Z (over 4 years ago)
- Default Branch: master
- Last Pushed: 2021-08-05T21:22:11.000Z (over 3 years ago)
- Last Synced: 2024-10-28T14:24:24.095Z (3 months ago)
- Topics: geut, hyperdrive, seeder, vault
- Language: JavaScript
- Homepage:
- Size: 1.83 MB
- Stars: 26
- Watchers: 3
- Forks: 3
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
- awesome-starred - geut/permanent-seeder - A super seeder for hyperdrives. By GEUT. (others)
README
# permanent-seeder
[](https://standardjs.com)
[](https://github.com/RichardLitt/standard-readme)
[](https://geutstudio.com)
```
npm i -g @geut/permanent-seeder
```
Alternatively you can `git clone` this repo and build it from source:
```bash
# install deps
npm i
npm run bootstrap
# build the dashboard
cd packages/dashboard
cp .env.example .env
npm run build
# start the seeder
cd ../cli
./bin/run stop
./bin/run start
```
First, create a base config file, you can tweak it later ;-)
```
$ permanent-seeder config:init
```
Then, start the Permanent Seeder daemon
```
$ permanent-seeder start
```
:rocket:
## Commands
```
$ permanent-seeder [COMMAND] [--OPTIONS]
```
### Config
```
$ permanent-seeder config:[init|get]
```
- `init`: creates the base config file for the Permanent Seeder. This is a `.toml` file that will live in `~/permanent-seeder/settings.toml`.
- `get`: returns the settings from the CLI. Useful when you are changing values and want to be sure they are pick up by the Permanent Seeder.
Default settings:
```toml
## Permanent seeder path (will be completed on config:init)
path = 'permanent-seeder'
## Enable stats recording
save_stats = true
## [optional] Preferred swarm port
# swarm_port =
## [optional] Api service settings
# [api]
# port = 3001
## [optional] HTTPS certificate and key
# [api.https]
# cert = "/path/to/cert.crt"
# key = "/path/to/cert.key"
## keys.endpoints = array of configs per endpoint
[[keys.endpoints]]
# Where to fetch keys
url = 'http://localhost:3000'
# Frequency between fetchs (in minutes)
frequency = 5
# Hook to parse response
hook = 'endpoint-hook.js'
## [optional] To add another endpoint, uncomment and complete next lines:
# [[keys.endpoints]]
# url =
# frequency =
# hook
```
### Start
```
$ permanent-seeder start
```
Bootstrap a Permanent Seeder instance that will keep up running in the background. If you change settings, you will need to call `start` command again.
### Status
```
$ permanent-seeder status
```
It will return instance status. If it is running and some basic stats.
### Stop
```
$ permanent-seeder stop
```
Stops the current instance.
### Dashboard
```
$ permanent-seeder dashboard
```
Opens the dashboard app in a browser. If you want to manually access the dashboard, it can be found in: `localhost:3001`
:warning: Note: The dashboard app runs in `http://localhost:3001`. If you deploy the Permanent Seeder on a server and wants to access the dashboard from the outside, you may need to setup a reverse proxy.
Sample nginx reverse proxy config
```
upstream dashboard-nodejs {
server 127.0.0.1:3001;
}
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
location / {
proxy_pass http://dashboard-nodejs;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
location /socket.io/ {
proxy_pass http://dashboard-nodejs;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
```
#### Environment Variables
We use the `.env.example` file as a placeholder for some environment variables that we use internally, like `PUBLIC_URL`.
If you build the Permanent Seeder from source then you may want to customize some things.
There are two variables that can be used:
- `PUBLIC_URL`: used to define the root path of the dashboard app. DEFAULT = '/'
- `REACT_APP_API_VERSION`: used to indicate the api version to use. DEFAULT = 'v1'
For **development** there are a few others variables that you can tweak, like the `SOCKET_URL` and `API_URL`. Checkout the full `.env.example`:
```env
REACT_APP_API_VERSION=v1
PUBLIC_URL=/
## Extend eslint config
EXTEND_ESLINT=true
## Optional ENVs only for development
#REACT_APP_SOCKET_URL=
#REACT_APP_API_URL=
```
### Key Management :key:
```
$ permanent-seeder key:[add|remove|remove-all]
```
- `add`: Insert a new key Permanent Seeder db, it will start downloading and seeding ASAP.
- `remove`: Removes a single key from the seeder db and also stops seeding it (e.g.: no more announcing to other peers)
- `remove-all`: Removes and unnanounce all the keys in the db.
### Logs
```
$ permanent-seeder logs --[live|all|error]
```
- `live`: like doing a `tail -f` of the logs.
- `all`: Show all the logs stored.
- `error`: Display only error logs.
### repl
```
$ permanent-seeder repl
```
Useful to inspect the Permanent Seeder under the hood. :microscope:
## Keys Endpoint
The Permanent Seeder can `fetch` keys from an external endpoint, i.e: perform a `GET` request against a particular endpoint. This can be useful if you maintain a service that stores hyperdrive's keys. If that is the case, then the Permament Seeder can fetch those keys regularly. You can think of this like a cron job.
Whilst, we internally expect an `Array<{url}>`, you can customize and parse the fetch response the way you need it.
To do this, you will need to modify `$HOME/permanent-seeder/endpoint-hook.js`.
That hook will be called after `fetch` the response.
You can also tweak the fetch `frequency` (defined in **minutes**) and the endpoint `url`. These options can be found in the `settings.toml` file:
```toml
[[keys.endpoints]]
url = "http://localhost:3000"
frequency = 5
hook = "$HOME/permanent-seeder/endpoint-hook.js"
```
### Delete Keys Endpoint
The Permanent Seeder also allows for another endpoint, this one for deleted keys. With this action the Permanent Seeder can fetch an array of keys similarly to what we defined previously for the keys endpoint (adding).
The setting looks like this:
```toml
[[keys.remove]]
url = "http://localhost:60620/deletes.json"
hook = "/Users/deka/permanent-seeder/endpoint-hook.js"
```
Again, we internally expect an `Array<{url}>`, but you can customize and parse the fetch response the way you need it.
To do this, you can tweak the hook response. By default the "delete keys endpoint" is using the same default hook used with the keys endpoint.
Finally, the fetch frequency is managed by the keys endpoint frequency. In other words, they run with the same frequency interval.
### Update Policy
Here are some rules that we follow to know when to update the keys.
Actions that result in update:
- add endpoint is defined and returns a list of keys. delete endpoint is not defined. :white_check_mark:
- add endpoint is defined and returns a list of keys. delete endpoint is defined and returns a list of keys. :white_check_mark:
Actions that does not result in update:
- add endpoint is defined and returns an empty list. Does not matter if delete endpoint is defined or not. :no_entry:
- add endpoint is defined but throws an error :no_entry:
- add endpoint is defined and returns a list of keys. delete endpoint is defined but throws an error. :no_entry:
## Design
The Permanent Seeder is a CLI tool that can starts a daemon which will [seed](https://en.wikipedia.org/wiki/Seeding_(computing)) [hyperdrive's](https://hypercore-protocol.org/#hyperdrive) keys that you pass into it.
Using the CLI you can add, remove keys, check the status and inspect logs.
It also contains a [`dashboard`](#dashboard) that you can use to have a visual reference of what is going on with your hyperdrives.
As you can see the project does a couple of things. To do this we decided to use a microservices based approach. We choose to use [moleculer](https://moleculer.services/) as the structural framework behind the Permanent Seeder. This enables multiples processes to communicate each other and at the same time each of these will have a single responsibility/scope. This also give us some room to scale things up if needed.
:sunglasses:
Below we define some internal API routes that can be used to obtain some information about the PS running instance as well as some basic drive queries.
### `GET /api//drives/:key`
Returns summary data associated to a specific drive `key`. Data may contain download information, peers info, seeding status and basic `index.json` information and version.
### `GET /api//drives/:key/size`
Returns only the download information.
### `GET /api//drives/:key/peers`
Returns only the peers information.
### `GET /api//drives/:key/stats`
Returns only information about the drive's files and its download status.
### `GET /api//drives/:key/info`
Returns only the `index.json` information part.
### `GET /api//drives/:key/seedingStatus`
Returns only the seeding status label.
### `GET /api//drives/:key/files/:file`
Download the file passed as argument from `key` drive.
### `GET /api/v1/stats/host`
Returns information about the host. CPU load, memory and directory usage.
### `GET /api/v1/stats/network`
Returns network information. Include things like online status, swarm information (hole-punching, bootstrapping info and network address), along with transferred bytes.
### `GET /api//raw/:key`
Returns the raw data associated with the seeded drive (`key`). The returned data may include various [lifecycle](packages/seeder/src/drive-events.md) events with their timestamps. This endpoint will return values as long as `save_stats` config property is enabled. See [config](#config).
### Filter by Version
You can also query data using the drive version.
### `GET /api//drives/:key/versions`
Returns all the versions saved for `key` drive
### `GET /api//drives/:key/:version/info`
Returns only the `info` field for this specific key and version
### `GET /api//drives/:key/:version/stats`
Returns only the `stats` (files) field for this specific key and version
### `GET /api//drives/:key/:version/files/:file`
Download the file passed as argument from a specific drive's checkout (`key`+`version`).
## Release
For releasing new versions we are using GitHub Actions :rocket: This is the process.
1. First, we create a new **release** branch: `git co -b username/release-${newVersion}`
2. Then, we need to create the new version. This new version will trigger a changelog update (using [chan](https://github.com/geut/chan)). To do this just run: `npm run release` It will prompt you what is the next version.
3. Create the **release pull-request**. The title should indicate this: `Release: newVersion`
- :warning: This is an important step, otherwise the publish workflow won't see the PR.
4. Finally, when the **release** PR is approved, a custom action will be triggered publishing the new version to npm, creating a new tag for it and creating a brand new github release :new: :package:
:bug: If you found an issue we encourage you to report it on [github](https://github.com/geut/permanent-seeder/issues). Please specify your OS and the actions to reproduce it.
:busts_in_silhouette: Ideas and contributions to the project are welcome. You must follow this [guideline](https://github.com/geut/permanent-seeder/blob/master/CONTRIBUTING.md).
## Built in collaboration with Liberate Science
## Acknowledgments
:clap: Many thanks to [@nicomanso](https://github.com/nicomanso) for offering some basic infrastructure that was used for early testing and [@krahser](https://github.com/krahser) for his outstanding docker contribution ([PR](https://github.com/geut/permanent-seeder/pull/105))
## License
MIT © A [**GEUT**](http://geutstudio.com/) project