{"id":18878568,"url":"https://github.com/verida/storage-node","last_synced_at":"2025-04-14T18:32:16.612Z","repository":{"id":38311000,"uuid":"404216580","full_name":"verida/storage-node","owner":"verida","description":null,"archived":false,"fork":false,"pushed_at":"2025-02-13T10:41:59.000Z","size":839,"stargazers_count":4,"open_issues_count":20,"forks_count":5,"subscribers_count":6,"default_branch":"develop","last_synced_at":"2025-03-28T07:22:09.534Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"isc","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/verida.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"license.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-08T05:00:03.000Z","updated_at":"2025-02-13T10:42:03.000Z","dependencies_parsed_at":"2023-12-01T01:26:10.649Z","dependency_job_id":"a3c6ca9e-64cd-490d-b1a5-8620c2d1c167","html_url":"https://github.com/verida/storage-node","commit_stats":null,"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/verida%2Fstorage-node","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/verida%2Fstorage-node/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/verida%2Fstorage-node/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/verida%2Fstorage-node/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/verida","download_url":"https://codeload.github.com/verida/storage-node/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248936878,"owners_count":21186118,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T06:28:07.975Z","updated_at":"2025-04-14T18:32:16.603Z","avatar_url":"https://github.com/verida.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Welcome to Verida Storage Node Server\n\nThis server acts as middleware between web applications built using the [Verida Datastore](https://developers.verida.io/docs/concepts/data-storage/) and the underlying databases storing user data.\n\nKey features:\n\n- Ensuring all API requests come from verified Verida network users (via user signed messages)\n- Managing database users, linking them to valid DID's\n- Managing permissions for individual databases\n- Adding a second layer of security by managing per-database ACL validation rules\n- Providing applications with user's database connection strings\n\n## How Authorization Works\n\nThis is the login flow:\n\n1. The Verida Account makes a request to the storage node API to obtain an auth JWT to be signed (`/auth/generateAuthJwt`). This prevents replay attacks.\n2. The Verida Account signs a consent message using their private key. This consent message proves the user wants to unlock a specific application context\n3. The Verida Account submits the signed authorization request (`/auth/authenticate`). Assuming the signed AuthJWT is valid, the storage node returns a refresh token and an access token\n4. The Verida Account can then use the access token to either; 1) make storage node requests (ie: create database) or 2) directly access CouchDB as an authenticated user (using `Bearer` token auth)\n5. When the access token expires, the Verida Account can use the refresh token to request a new access token (`/auth/connect`)\n6. If a refresh token is close to expiry, the Verida Account can use the active refresh token to obtain a new refresh token (`/auth/regenerateRefreshToken`)\n\nWhen a Verida Account authenticates, it can designate an `authenticate` requst to be linked to a particular device by specifying the `deviceId` in the request.\n\nThis allows a specific device to be linked to a refresh token. A call to `/auth/invalidateDeviceId` can be used to invalidate any refresh tokens linked to the specified `deviceId`. This allows the Verida Vault to remotely log out an application that previously logged in.\n\nNote: This only invalidates the refresh token. The access token will remain valid until it expires. It's for this reason that access tokens are configured to have a short expiry (5 minutes by default). CouchDB does not support manually invalidating access tokens, so we have to take this timeout approach to invalidation.\n\n## Usage\n\n```bash\nyarn install\nyarn build\nyarn serve\n```\n\n## Configuration\n\nA `sample.env` is included. Copy this to `.env` and update the configuration:\n\n- `VERIDA_NETWORK`: Verida network to use. See https://developers.verida.network/docs/infrastructure/networks for valid networks. (ie: `banksia`)\n- `DB_PROTOCOL`: Protocol to use when connecting to CouchDB (`http` or `https`).\n- `DB_USER`: Username of CouchDB Admin (has access to create users and databases).\n- `DB_PASS`: Password of CouchDB Admin.\n- `DB_HOST`: Hostname of CouchDB Admin.\n- `DB_PORT`: Port of CouchDB server (`5984`).\n- `DB_REPLICATION_USER`: Replication username (for replicating data to other nodes). MUST be set to something random. MUST not change once the node is operational.\n- `DB_REPLICATION_PASS`: Replication password (for replicating data to other nodes). MUST be set to something random. MUST not change once the node is operational.\n- `DB_REJECT_UNAUTHORIZED_SSL`: Boolean indicating if unauthorized SSL certificates should be rejected (`true` or `false`). Defaults to `false` for development testing. Must be `true` for production environments otherwise SSL certificates won't be verified.\n- `DB_PUBLIC_USER`: Alphanumeric string for a public database user. These credentials can be requested by anyone and provide access to all databases where the permissions have been set to `public`.\n- `DB_PUBLIC_PASS`: Alphanumeric string for a public database password.\n- `DB_REFRESH_TOKENS`: Internal CouchDB database that stores refresh tokens (ie: `verida_refresh_tokens`)\n- `ACCESS_JWT_SIGN_PK`: The access token private key. The base64 version of this must be specified in the CouchDB configuration under `jwt_keys/hmac:_default`\n- `REFRESH_JWT_SIGN_PK`: The refresh token private key\n- `DB_PROTOCOL_INTERNAL`: Internal database protocol (`http` or `https`).\n- `DB_HOST_INTERNAL`: Internal database hostname (ie: `localhost`)\n- `DB_PORT_INTERNAL`: Internal database port (ie: `5984`)\n- `DB_PROTOCOL_INTERNAL`: External database protocol (`http` or `https`).\n- `DB_HOST_INTERNAL`: External database hostname (ie: `mydomain.com`)\n- `DB_PORT_INTERNAL`: External database port (ie: `5984`)\n- `ENDPOINT_URI`: The public URI of this storage node server (Will match what is stored in DID Documents). Note: Must include the port and have NO trailing slash. (ie: `\"http://localhost:5000\"`)\n- `VDA_PRIVATE_KEY`: Verida network private key as a hex string. Including leading 0x. This is used to sign server responses and in the future, prove VDA tokens are staked for this node. (ie: `0xaaaabbbb...`)\n- `MAX_USERS`: Maximum number of users supported by this node (ie: `10000`)\n- `PORT`: Port this server runs on (ie: `5151`)\n\n\n### Setting up environment variables on Windows\n\n* On a powershell execute the following ( replica of `.env` )\n```bash\n$env:DID_CACHE_DURATION=3600\n$env:DB_PROTOCOL=\"http\"\n$env:DB_USER=\"admin\"\n$env:DB_PASS=\"admin\"\n$env:DB_HOST=\"localhost\"\n$env:DB_PORT=5984\n$env:DB_REJECT_UNAUTHORIZED_SSL=false\n$env:DB_PUBLIC_USER=\"784c2n780c9cn0789\"\n$env:DB_PUBLIC_PASS=\"784c2n780c9cn0789\"\n```\n\n## CouchDB configuration\n\n- CORS must be enabled so that database requests can come from any domain name\n- A valid user must be enforced for security reasons\n\n[Ensure `{chttpd_auth, jwt_authentication_handler}` is added to the list of the active `chttpd/authentication_handlers`](https://docs.couchdb.org/en/stable/api/server/authn.html?highlight=jwt#jwt-authentication)\n\n\n```\n[couchdb]\nsingle_node=true\n\n[chttpd]\nauthentication_handlers = {chttpd_auth, jwt_authentication_handler}, {chttpd_auth, cookie_authentication_handler}, {chttpd_auth, default_authentication_handler}\nenable_cors = true\n\n[chttpd_auth]\nrequire_valid_user = true\n\n[jwt_auth]\nrequired_claims = exp\n\n[jwt_keys]\nhmac:_default = \u003cbase64 secret key\u003e\n\n[cors]\norigins = *\ncredentials = true\nmethods = GET, PUT, POST, HEAD, DELETE\nheaders = accept, authorization, content-type, origin, referer, x-csrf-token\n```\n\nThe `hmac:_default` key is a base64 encoded representation of the access token JWT private key\n\n## Generating JWT key\n\nNote: A secret key (string) suitable for `jwt_keys` can be base64 encoded with the following:\n\n```js\nconst secretKey = 'secretKey'\nconst encodedKey = Buffer.from(secretKey).toString('base64')\n```\n\nYou can generate the base64 encoced string with:\n\n```bash\nnode src/base64key.js \u003cyourkey\u003e\n```\n\nThis can be tested via curl:\n\n```bash\ncurl -H \"Host: localhost:5984\" \\\n -H \"accept: application/json, text/plain, */*\" \\\n -H \"authorization: Bearer \u003cbearer_token\u003e\" \\\n  \"http://localhost:5984/_session\"\n```\n\nWhere:\n\n- `bearer_token` - A bearer token generated via the `test/jwt` unit test\n- `localhost` - Replace this with the hostname of the server being tested\n\n## Docker\n\nYou can spin up storage node API on your machine with Docker:\n```shell\ndocker run --init --env-file=.env verida/storage-node:latest\n```\n\n### Deploying a new Docker Image to Docker Hub\n\nNote that this uses the experimental `buildx` command to build both AMD64 (Intel/AMD servers) and ARM64 (Mac) images.\n\n* Login (details in BitWarden)\n```\ndocker buildx build --platform linux/amd64,linux/arm64 --push -t verida/storage-node:latest .\n```\n\n## Tests\n\nRun tests with `yarn run tests`\n\nYou will need to update `/test/config.js` with at least the `VDA_PRIVATE_KEY`, `DID_CLIENT_CONFIG.web3Config.privateKey` \nand possibily `ENDPOINTS` and `SERVER_URL`.\n\nCommon issues when running tests:\n\n1. `Bad key`: The key in CouchDB configuration for `jwt_keys/hmac:_default` is not a valid Base64 encoded key\n2. `HMAC error`: The key in CouchDB configuration for `jwt_keys/hmac:_default` does not match `ACCESS_JWT_SIGN_PK` in `.env`\n\n\n## Testing a deployed node\n\nTo test a deployed node, do the following\n\n* Modify `test/config.js` with the correct endpoint URLs\n* Run `yarn test test/server.js`\n\n# Production Infrastructure Notes\n\nWhen deploying behind a HTTP load balancer it is important to make sure it doesn't close the connection during a long-poll call. The server will keep pushing data through this connection, but some load balancers (eg the Google Cloud loadbalancer) will close anyway. In these case configure the load balancer with a long timeout (we recommend 3600 seconds).\n\n# Infrastructure requirements\n\nThe Verida team is currently operating nodes with the following hardware requirements:\n\n1. 16GB RAM\n2. 4vCPUs\n3. 1TB storage (20,000 storage slots)\n\nThese cost approx $US250 / month on AWS / GCP / Azure.\n\nIdeally nodes will have burstable CPU usage as the number of connected users is CPU bound and can fluxuate depending on active demand from end users.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fverida%2Fstorage-node","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fverida%2Fstorage-node","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fverida%2Fstorage-node/lists"}