{"id":13553424,"url":"https://github.com/geut/permanent-seeder","last_synced_at":"2025-04-28T12:29:01.335Z","repository":{"id":48117791,"uuid":"272820638","full_name":"geut/permanent-seeder","owner":"geut","description":"A super seeder for hyperdrives. By GEUT.","archived":false,"fork":false,"pushed_at":"2021-08-05T21:22:11.000Z","size":1923,"stargazers_count":26,"open_issues_count":5,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-13T03:18:11.051Z","etag":null,"topics":["geut","hyperdrive","seeder","vault"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/geut.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-16T22:00:14.000Z","updated_at":"2023-06-14T20:49:58.000Z","dependencies_parsed_at":"2022-08-12T19:00:41.480Z","dependency_job_id":null,"html_url":"https://github.com/geut/permanent-seeder","commit_stats":null,"previous_names":[],"tags_count":35,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geut%2Fpermanent-seeder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geut%2Fpermanent-seeder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geut%2Fpermanent-seeder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/geut%2Fpermanent-seeder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/geut","download_url":"https://codeload.github.com/geut/permanent-seeder/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251313120,"owners_count":21569362,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["geut","hyperdrive","seeder","vault"],"created_at":"2024-08-01T12:02:24.394Z","updated_at":"2025-04-28T12:29:01.288Z","avatar_url":"https://github.com/geut.png","language":"JavaScript","funding_links":[],"categories":["JavaScript","others"],"sub_categories":[],"readme":"# permanent-seeder\n\n![CI Status](https://github.com/geut/permanent-seeder/workflows/CI/badge.svg?style=flat-square)\n\n![npm version](https://img.shields.io/npm/v/@geut/permanent-seeder?color=brightgreen\u0026style=flat-square)\n\n[![JavaScript Style Guide](https://img.shields.io/badge/code_style-standard-brightgreen.svg?style=flat-square)](https://standardjs.com)\n[![standard-readme compliant](https://img.shields.io/badge/readme%20style-standard-brightgreen.svg?style=flat-square)](https://github.com/RichardLitt/standard-readme)\n\n[![Made by GEUT](https://img.shields.io/badge/Made%20By-GEUT-4f5186?style=for-the-badge\u0026link=https://geutstudio.com\u0026labelColor=white\u0026logo=data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wAARCABAAEADASIAAhEBAxEB/8QAGwAAAgIDAQAAAAAAAAAAAAAABAYDBQACBwH/xAA0EAACAQMBBAcGBgMAAAAAAAABAgMABBEFBhIhQRMiMVFhgcEUIzJxkbFCUmKh0fAkcuH/xAAYAQADAQEAAAAAAAAAAAAAAAABAwQCAP/EACARAAMAAwACAgMAAAAAAAAAAAABAgMRIRIxBEEiM1H/2gAMAwEAAhEDEQA/AOgVlau6xoXdgqqMkk8AKV9U2oYs0WngBRw6VhxPyFamXXoDeiz1PUbmzuujQIUKgjIqGLXnz72FSO9TikfVbi6uXWSSaWRuzixNBx3VzCepNIvgTw+hpjwv+iGr3tM6xa30F2PdP1uangRRNc70fUbi4JLIVaPskXgM/wA076Ze+2W+WwJF4MPWlNaemajI2/GvYbWVlZQHCptZqLNKLGJsKoDSY5nkKorKzlvrlYIRlm5nsA7zWX8pnv55SfikJ/emPZGDcs7m6CguTuL5DPrVf64Me2F2mzNhAg6ZTO/MsSB9BW15s1pt1GVEPRHvQ+hqbTNT9sZ0kCpIOIA5ij5ZEijaSRgqqMkmpVkb7sMuWtoV73S49L3I4B7kjq57c881BZ6vFpuoKjq7dIvYBw8PtUOqX1xcSxoJXw8mQuewVW3vX1eFR+Fcn96OLVvpFzz8kM020kp4QwIvixzVpot5Je2bSTEFw5HAY7qUKadnIymm7x/G5I+3pTskzM8G4rqq6JGpI8E1wi8HR2H0NT7P6rcRKUEzYR9/czgEf0VabV2JgvhdKPdzdvg399aVG37K4Esfw/3hTU1S2NpNrSHqax9q/wAzTm3lY5KA4ZTQl2mo9CWljncL+cnA+tVVhqeSGt5mik5qDg/9o+XVb6aFonuDusMHqjP2qavjbfGTPX3xgTstrm4uGDSEYVV+woWPMKy3dzwd+JHcOQrdkgtyZpXJb87nJ8qqr68a7cKgIjB4DmadGNQjohs9i1C66Xqtvbx+EjIp10jaOMLBaPasDwRTGc5PyNJ1rb9EN5/jP7U17KaaZJvbpV6icI88z3+VG0vH8ipJJ8Ga8tIr22eCYZVh5g94pC1TTJtPmMU67yH4XxwYV0So54IriIxzRrIh7QwzSIyOTbWzlElkCcxtjwNedHeKMCVseDmnq72UgkJa1maL9LDeH81XvspfA9WSBh/sR6U9XD+zDQp+yTSNmR/MnJomG3SLiBlu80zQ7JXTH31xEg/Tlj6Vb2OzljaEO6meQc5OweVc8koOmUGjaFLfuss4MdsOOewv8v5p0ijSGNY41CoowAOQrbsr2p7t0zSWj//Z)](https://geutstudio.com)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg alt=\"Permanent Seeder logo, a  beautiful seed surrounded with a round and exquisite black border\" src=\"packages/dashboard/public/permanent-seeder-192.png\"/\u003e\n\u003c/p\u003e\n\n## \u003ca name=\"install\"\u003e\u003c/a\u003e Install\n\n```\nnpm i -g @geut/permanent-seeder\n```\n\nAlternatively you can `git clone` this repo and build it from source:\n\n```bash\n# install deps\nnpm i\nnpm run bootstrap\n\n# build the dashboard\ncd packages/dashboard\ncp .env.example .env\nnpm run build\n\n# start the seeder\ncd ../cli\n./bin/run stop\n./bin/run start\n\n```\n\n## \u003ca name=\"usage\"\u003e\u003c/a\u003e Usage\n\nFirst, create a base config file, you can tweak it later ;-)\n```\n$ permanent-seeder config:init\n```\n\nThen, start the Permanent Seeder daemon\n```\n$ permanent-seeder start\n```\n:rocket:\n\n## \u003ca name=\"commands\"\u003e\u003c/a\u003e Commands\n```\n$ permanent-seeder [COMMAND] [--OPTIONS]\n```\n### Config\n```\n$ permanent-seeder config:[init|get]\n```\n- `init`: creates the base config file for the Permanent Seeder. This is a `.toml` file that will live in `~/permanent-seeder/settings.toml`.\n- `get`: returns the settings from the CLI. Useful when you are changing values and want to be sure they are pick up by the Permanent Seeder.\n\nDefault settings:\n```toml\n## Permanent seeder path (will be completed on config:init)\npath = 'permanent-seeder'\n\n## Enable stats recording\nsave_stats = true\n\n## [optional] Preferred swarm port\n# swarm_port =\n\n## [optional] Api service settings\n# [api]\n# port = 3001\n\n## [optional] HTTPS certificate and key\n# [api.https]\n# cert = \"/path/to/cert.crt\"\n# key = \"/path/to/cert.key\"\n\n## keys.endpoints = array of configs per endpoint\n[[keys.endpoints]]\n\n  # Where to fetch keys\n  url = 'http://localhost:3000'\n\n  # Frequency between fetchs (in minutes)\n  frequency = 5\n\n  # Hook to parse response\n  hook = 'endpoint-hook.js'\n\n## [optional] To add another endpoint, uncomment and complete next lines:\n# [[keys.endpoints]]\n#   url =\n#   frequency =\n#   hook\n```\n\n### Start\n\n```\n$ permanent-seeder start\n```\nBootstrap a Permanent Seeder instance that will keep up running in the background. If you change settings, you will need to call `start` command again.\n\n### Status\n```\n$ permanent-seeder status\n```\nIt will return instance status. If it is running and some basic stats.\n\n### Stop\n```\n$ permanent-seeder stop\n```\nStops the current instance.\n\n### Dashboard\n```\n$ permanent-seeder dashboard\n```\nOpens the dashboard app in a browser. If you want to manually access the dashboard, it can be found in: `localhost:3001`\n\n\u003cimg src=\"./Dashboard.png\" width=\"800px\" height=\"auto\" style=\"border:2px solid #fff; box-shadow: 10px 10px 5px #ccc\"\u003e\n\n:warning: Note: The dashboard app runs in `http://localhost:3001`. If you deploy the Permanent Seeder on a server and wants to access the dashboard from the outside, you may need to setup a reverse proxy.\n\n\u003cdetails\u003e\n  \u003csummary\u003eSample nginx reverse proxy config\u003c/summary\u003e\n\n```\nupstream dashboard-nodejs {\n        server  127.0.0.1:3001;\n}\n\nserver {\n\tlisten 80 default_server;\n\tlisten [::]:80 default_server;\n\n\tserver_name _;\n\n        location / {\n                proxy_pass              http://dashboard-nodejs;\n                proxy_next_upstream     error timeout invalid_header http_500 http_502 http_503 http_504;\n                proxy_redirect          off;\n                proxy_buffering         off;\n\n                proxy_set_header        Host                    $host;\n                proxy_set_header        X-Real-IP               $remote_addr;\n                proxy_set_header        X-Forwarded-For         $proxy_add_x_forwarded_for;\n                proxy_set_header        X-Forwarded-Proto       $scheme;\n        }\n\n        location /socket.io/ {\n                proxy_pass              http://dashboard-nodejs;\n                proxy_redirect          off;\n\n                proxy_http_version      1.1;\n\n                proxy_set_header        Upgrade                 $http_upgrade;\n                proxy_set_header        Connection              \"upgrade\";\n                proxy_set_header        Host                    $host;\n                proxy_set_header        X-Real-IP               $remote_addr;\n                proxy_set_header        X-Forwarded-For         $proxy_add_x_forwarded_for;\n        }\n}\n```\n\u003c/details\u003e\n\n\n#### Environment Variables\n\nWe use the `.env.example` file as a placeholder for some environment variables that we use internally, like `PUBLIC_URL`.\n\nIf you build the Permanent Seeder from source then you may want to customize some things.\n\nThere are two variables that can be used:\n- `PUBLIC_URL`: used to define the root path of the dashboard app. DEFAULT = '/'\n- `REACT_APP_API_VERSION`: used to indicate the api version to use. DEFAULT = 'v1'\n\nFor **development** there are a few others variables that you can tweak, like the `SOCKET_URL` and `API_URL`. Checkout the full `.env.example`:\n\n```env\nREACT_APP_API_VERSION=v1\nPUBLIC_URL=/\n\n## Extend eslint config\nEXTEND_ESLINT=true\n\n## Optional ENVs only for development\n#REACT_APP_SOCKET_URL=\n#REACT_APP_API_URL=\n```\n\n### Key Management :key:\n```\n$ permanent-seeder key:[add|remove|remove-all]\n```\n- `add`: Insert a new key Permanent Seeder db, it will start downloading and seeding ASAP.\n- `remove`: Removes a single key from the seeder db and also stops seeding it (e.g.: no more announcing to other peers)\n- `remove-all`: Removes and unnanounce all the keys in the db.\n\n### Logs\n\n```\n$ permanent-seeder logs --[live|all|error]\n```\n- `live`: like doing a `tail -f` of the logs.\n- `all`: Show all the logs stored.\n- `error`: Display only error logs.\n\n### repl\n```\n$ permanent-seeder repl\n```\nUseful to inspect the Permanent Seeder under the hood. :microscope:\n\n## Keys Endpoint\n\nThe Permanent Seeder can `fetch` keys from an external endpoint, i.e: perform a `GET` request against a particular endpoint. This can be useful if you maintain a service that stores hyperdrive's keys. If that is the case, then the Permament Seeder can fetch those keys regularly. You can think of this like a cron job.\n\nWhilst, we internally expect an `Array\u003c{url}\u003e`, you can customize and parse the fetch response the way you need it.\n\nTo do this, you will need to modify `$HOME/permanent-seeder/endpoint-hook.js`.\n\nThat hook will be called after `fetch` the response.\n\nYou can also tweak the fetch `frequency` (defined in **minutes**) and the endpoint `url`. These options can be found in the `settings.toml` file:\n```toml\n[[keys.endpoints]]\nurl = \"http://localhost:3000\"\nfrequency = 5\nhook = \"$HOME/permanent-seeder/endpoint-hook.js\"\n```\n\n### Delete Keys Endpoint\n\nThe Permanent Seeder also allows for another endpoint, this one for deleted keys. With this action the Permanent Seeder can fetch an array of keys similarly to what we defined previously for the keys endpoint (adding).\n\nThe setting looks like this:\n```toml\n [[keys.remove]]\n url = \"http://localhost:60620/deletes.json\"\n hook = \"/Users/deka/permanent-seeder/endpoint-hook.js\"\n```\n\nAgain, we internally expect an `Array\u003c{url}\u003e`, but you can customize and parse the fetch response the way you need it.\n\nTo do this, you can tweak the hook response. By default the \"delete keys endpoint\" is using the same default hook used with the keys endpoint.\n\nFinally, the fetch frequency is managed by the keys endpoint frequency. In other words, they run with the same frequency interval.\n\n### Update Policy\n\nHere are some rules that we follow to know when to update the keys.\n\nActions that result in update:\n- add endpoint is defined and returns a list of keys. delete endpoint is not defined. :white_check_mark:\n- add endpoint is defined and returns a list of keys. delete endpoint is defined and returns a list of keys. :white_check_mark:\n\nActions that does not result in update:\n- add endpoint is defined and returns an empty list. Does not matter if delete endpoint is defined or not. :no_entry:\n- add endpoint is defined but throws an error :no_entry:\n- add endpoint is defined and returns a list of keys. delete endpoint is defined but throws an error. :no_entry:\n\n\n## Design\n\nThe Permanent Seeder is a CLI tool that can starts a daemon which will [seed](https://en.wikipedia.org/wiki/Seeding_(computing)) [hyperdrive's](https://hypercore-protocol.org/#hyperdrive) keys that you pass into it.\nUsing the CLI you can add, remove keys, check the status and inspect logs.\n\nIt also contains a [`dashboard`](#dashboard) that you can use to have a visual reference of what is going on with your hyperdrives.\n\nAs you can see the project does a couple of things. To do this we decided to use a microservices based approach. We choose to use [moleculer](https://moleculer.services/) as the structural framework behind the Permanent Seeder. This enables multiples processes to communicate each other and at the same time each of these will have a single responsibility/scope. This also give us some room to scale things up if needed.\n:sunglasses:\n\n## \u003ca name=\"routes\"\u003e\u003c/a\u003e Routes\n\nBelow we define some internal API routes that can be used to obtain some information about the PS running instance as well as some basic drive queries.\n\n### `GET /api/\u003cAPI_VERSION\u003e/drives/:key`\n\nReturns summary data associated to a specific drive `key`. Data may contain download information, peers info, seeding status and basic `index.json` information and version.\n\n### `GET /api/\u003cAPI_VERSION\u003e/drives/:key/size`\n\nReturns only the download information.\n\n### `GET /api/\u003cAPI_VERSION\u003e/drives/:key/peers`\n\nReturns only the peers information.\n\n### `GET /api/\u003cAPI_VERSION\u003e/drives/:key/stats`\n\nReturns only information about the drive's files and its download status.\n\n### `GET /api/\u003cAPI_VERSION\u003e/drives/:key/info`\n\nReturns only the `index.json` information part.\n\n### `GET /api/\u003cAPI_VERSION\u003e/drives/:key/seedingStatus`\n\nReturns only the seeding status label.\n\n### `GET /api/\u003cAPI_VERSION\u003e/drives/:key/files/:file`\n\nDownload the file passed as argument from `key` drive.\n\n### `GET /api/v1/stats/host`\n\nReturns information about the host. CPU load, memory and directory usage.\n\n### `GET /api/v1/stats/network`\n\nReturns network information. Include things like online status, swarm information (hole-punching, bootstrapping info and network address), along with transferred bytes.\n\n### `GET /api/\u003cAPI_VERSION\u003e/raw/:key`\n\nReturns the raw data associated with the seeded drive (`key`). The returned data may include various [lifecycle](packages/seeder/src/drive-events.md) events with their timestamps. This endpoint will return values as long as `save_stats` config property is enabled. See [config](#config).\n\n### Filter by Version\n\nYou can also query data using the drive version.\n\n### `GET /api/\u003cAPI_VERSION\u003e/drives/:key/versions`\n\nReturns all the versions saved for `key` drive\n\n### `GET /api/\u003cAPI_VERSION\u003e/drives/:key/:version/info`\n\nReturns only the `info` field for this specific key and version\n\n### `GET /api/\u003cAPI_VERSION\u003e/drives/:key/:version/stats`\n\nReturns only the `stats` (files) field for this specific key and version\n\n### `GET /api/\u003cAPI_VERSION\u003e/drives/:key/:version/files/:file`\n\nDownload the file passed as argument from a specific drive's checkout (`key`+`version`).\n\n\n## Release\n\nFor releasing new versions we are using GitHub Actions :rocket: This is the process.\n\n1. First, we create a new **release** branch: `git co -b username/release-${newVersion}`\n2. Then, we need to create the new version. This new version will trigger a changelog update (using [chan](https://github.com/geut/chan)). To do this just run: `npm run release` It will prompt you what is the next version.\n3. Create the **release pull-request**. The title should indicate this: `Release: newVersion`\n    - :warning: This is an important step, otherwise the publish workflow won't see the PR.\n4. Finally, when the **release** PR is approved, a custom action will be triggered publishing the new version to npm, creating a new tag for it and creating a brand new github release :new: :package:\n\n## \u003ca name=\"issues\"\u003e\u003c/a\u003e Issues\n\n:bug: If you found an issue we encourage you to report it on [github](https://github.com/geut/permanent-seeder/issues). Please specify your OS and the actions to reproduce it.\n\n## \u003ca name=\"contribute\"\u003e\u003c/a\u003e Contributing\n\n:busts_in_silhouette: Ideas and contributions to the project are welcome. You must follow this [guideline](https://github.com/geut/permanent-seeder/blob/master/CONTRIBUTING.md).\n\n## Built in collaboration with Liberate Science\n\n\u003ca href=\"https://libscie.org\" rel=\"nofollow\"\u003e\n\u003cimg src=\"https://github.com/libscie.png\" alt=\"Liberate Science\" width=\"200px\" style=\"max-width:100%;\"\u003e\n\u003c/a\u003e\n\n## Acknowledgments\n\n:clap: Many thanks to [@nicomanso](https://github.com/nicomanso) for offering some basic infrastructure that was used for early testing and [@krahser](https://github.com/krahser) for his outstanding docker contribution ([PR](https://github.com/geut/permanent-seeder/pull/105))\n\n## License\n\nMIT © A [**GEUT**](http://geutstudio.com/) project\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeut%2Fpermanent-seeder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgeut%2Fpermanent-seeder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeut%2Fpermanent-seeder/lists"}