{"id":20138332,"url":"https://github.com/pinax-network/substreams-sink-webhook","last_synced_at":"2025-03-02T23:14:51.048Z","repository":{"id":171494159,"uuid":"637834914","full_name":"pinax-network/substreams-sink-webhook","owner":"pinax-network","description":"Substreams Sink Webhook","archived":false,"fork":false,"pushed_at":"2024-02-21T01:53:09.000Z","size":409,"stargazers_count":2,"open_issues_count":8,"forks_count":0,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-01-13T09:46:46.797Z","etag":null,"topics":["streamingfast","substreams","thegraph","webhook"],"latest_commit_sha":null,"homepage":"https://github.com/pinax-network/substreams-sink-webhook/pkgs/container/substreams-sink-webhook","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pinax-network.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-08T14:05:06.000Z","updated_at":"2024-08-08T19:45:05.000Z","dependencies_parsed_at":null,"dependency_job_id":"74ddd986-20b2-4d3c-af33-abada8f2f4d4","html_url":"https://github.com/pinax-network/substreams-sink-webhook","commit_stats":null,"previous_names":["pinax-network/substreams-sink-webhook"],"tags_count":28,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pinax-network%2Fsubstreams-sink-webhook","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pinax-network%2Fsubstreams-sink-webhook/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pinax-network%2Fsubstreams-sink-webhook/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pinax-network%2Fsubstreams-sink-webhook/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pinax-network","download_url":"https://codeload.github.com/pinax-network/substreams-sink-webhook/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241582552,"owners_count":19985845,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["streamingfast","substreams","thegraph","webhook"],"created_at":"2024-11-13T21:37:27.796Z","updated_at":"2025-03-02T23:14:51.024Z","avatar_url":"https://github.com/pinax-network.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# [`Substreams`](https://substreams.streamingfast.io/) Sink Webhook\n\n\u003e `substreams-sink-webhook` is a tool that allows developers to pipe data extracted from a blockchain to Webhook.\n\n## HTTP Server examples\n\n- [`Bun`](/examples/bun) - https://bun.sh/\n- [`Express`](/examples/express) - https://expressjs.com/\n- [`node:http`](/examples/node) - https://nodejs.org/api/http.html\n- [POST request](/examples/http)\n\n## 📖 References\n\n- [**Substreams** documentation](https://substreams.streamingfast.io/)\n- TweetNaCl\n  - [TweetNaCl: a crypto library in 100 tweets](http://tweetnacl.cr.yp.to/)\n  - [TweetNaCl.js](https://tweetnacl.js.org/)\n  - [TweetNaclSharp](https://github.com/XeroXP/TweetNaclSharp)\n- Discord Webhooks\n  - [interactions-and-bot-users](https://discord.com/developers/docs/interactions/receiving-and-responding#interactions-and-bot-users)\n  - [security-and-authorization](https://discord.com/developers/docs/interactions/receiving-and-responding#security-and-authorization)\n\n## POST Message\n\nThe POST message will be a JSON object with the following structure:\n\n**headers**\n\n```http\nPOST http://localhost:3000 HTTP/1.1\ncontent-type: application/json\nx-signature-ed25519: 8f01c66ccda5b987c43d913290419572ea586dbef2077fa166c4a84797e1d2c76b305bc67ed43efb1fc841562620a61cb59c4d8a13de689a2e98ead19190f80c\nx-signature-timestamp: 1707776632\n```\n\n**body**\n\n\u003e The body will be a flatten JSON string with the following structure:\n\n```json\n{\n  \"status\": 200,\n  \"cursor\": \"OJGbpO9ZnZcwvxW38_FO8KWwLpcyA1lrUQPgKRFL04Py8yCW35v1VTB1O0-Elami3RztQlOp2tmcHC9y9ZQFuoDrxLpj6yU-FXorwoHr_OfqLPumMQwTJ-hgWeuKYNLeWDjTagn4ersEtNGzbvLaY0UxZZUhK2G62z1VptdXJfEWuiJmyjmrIZrRhK-WoNAS_rEkQ7L1xCmhDzJ4K0dTPcSDNPKZuDR2\",\n  \"session\": {\n    \"traceId\": \"3cbb0a1c772a47a72995d95f4c6d2cff\",\n    \"resolvedStartBlock\": 53448515\n  },\n  \"clock\": {\n    \"timestamp\": \"2024-02-12T22:23:51.000Z\",\n    \"number\": 53448530,\n    \"id\": \"f843bc26cea0cbd50b09699546a8a97de6a1727646c17a857c5d8d868fc26142\"\n  },\n  \"manifest\": {\n    \"substreamsEndpoint\": \"https://polygon.substreams.pinax.network:443\",\n    \"chain\": \"polygon\",\n    \"finalBlockOnly\": \"false\",\n    \"moduleName\": \"map_blocks\",\n    \"type\": \"sf.substreams.v1.Clock\",\n    \"moduleHash\": \"44c506941d5f30db6cca01692624395d1ac40cd1\"\n  },\n  \"data\": {\n    ...\n  }\n}\n```\n\n## Generate Ed25519 public \u0026 private key pair\n\n```bash\n$ bunx substreams-sink-webhook keypair\n{\n  \"publicKey\": \"36a89085d54d866c60ecccc2bf332d1c0dd5f1a810af175b1cfb7ff9e64b67d6\",\n  \"privateKey\": \"67603675f8160b4e4ca67770eaf7df797f3a9617665a84ec3e9baf92c403fb4f\"\n}\n```\nor using `curl`\n\n```bash\n$ curl http://localhost:9102/keypair\n```\n\n## Validate Ed25519 signature\n\n```typescript\nimport nacl from \"tweetnacl\";\n\n// ...HTTP server\nconst PUBLIC_KEY = \"APPLICATION_PUBLIC_KEY\";\n\n// get headers and body from POST request\nconst signature = request.headers.get(\"x-signature-ed25519\");\nconst timestamp = request.headers.get(\"x-signature-timestamp\");\nconst body = await request.text();\n\n// validate signature using public key\nconst isVerified = nacl.sign.detached.verify(\n  Buffer.from(timestamp + body),\n  Buffer.from(signature, \"hex\"),\n  Buffer.from(PUBLIC_KEY, \"hex\")\n);\n\nif (!isVerified) {\n  return new Response(\"invalid request signature\", { status: 401 });\n}\n```\n\n## `.env` Environment variables\n\n```env\n# Webhook\nPORT=9102\nWEBHOOK_URL=http://127.0.0.1:3000\n\n# Get Substreams API Key\n# https://app.pinax.network\n# https://app.streamingfast.io/\nSUBSTREAMS_API_KEY=\u003cSubstreams API Token @ https://pinax.network\u003e\n\n# Substreams Package (*.spkg)\nMANIFEST=https://github.com/pinax-network/substreams/releases/download/blocks-v0.1.0/blocks-v0.1.0.spkg\nMODULE_NAME=map_blocks\nSTART_BLOCK=-10\nPRODUCTION_MODE=true\n\n# Webhook (Optional)\nPRIVATE_KEY=\u003cEd25519 Private Key\u003e\nMAXIMUM_ATTEMPTS=100\nVERBOSE=true\n```\n\n## Help\n\n```\n$ substreams-sink-webhook --help\n\nUsage: substreams-sink-webhook run [options]\n\nSubstreams Sink Webhook\n\nOptions:\n  -e --substreams-endpoint \u003cstring\u003e    Substreams gRPC endpoint to stream data from (env: SUBSTREAMS_ENDPOINT)\n  --manifest \u003cstring\u003e                  URL of Substreams package (env: MANIFEST)\n  --module-name \u003cstring\u003e               Name of the output module (declared in the manifest) (env: MODULE_NAME)\n  -s --start-block \u003cint\u003e               Start block to stream from (defaults to -1, which means the initialBlock of the first module you are streaming) (default: \"-1\", env: START_BLOCK)\n  -t --stop-block \u003cint\u003e                Stop block to end stream at, inclusively (env: STOP_BLOCK)\n  -p, --params \u003cstring...\u003e             Set a params for parameterizable modules. Can be specified multiple times. (ex: -p module1=valA -p module2=valX\u0026valY) (default: [], env: PARAMS)\n  --substreams-api-key \u003cstring\u003e        API key for the Substream endpoint (env: SUBSTREAMS_API_KEY)\n  --delay-before-start \u003cint\u003e           Delay (ms) before starting Substreams (default: 0, env: DELAY_BEFORE_START)\n  --cursor-path \u003cstring\u003e               File path or URL to cursor lock file (default: \"cursor.lock\", env: CURSOR_PATH)\n  --http-cursor-auth \u003cstring\u003e          Basic auth credentials for http cursor (ex: username:password) (env: HTTP_CURSOR_AUTH)\n  --production-mode \u003cboolean\u003e          Enable production mode, allows cached Substreams data if available (default: \"false\", env: PRODUCTION_MODE)\n  --inactivity-seconds \u003cint\u003e           If set, the sink will stop when inactive for over a certain amount of seconds (default: 300, env: INACTIVITY_SECONDS)\n  --hostname \u003cstring\u003e                  The process will listen on this hostname for any HTTP and Prometheus metrics requests (default: \"localhost\", env: HOSTNAME)\n  --port \u003cint\u003e                         The process will listen on this port for any HTTP and Prometheus metrics requests (default: 9102, env: PORT)\n  --metrics-labels [string...]         To apply generic labels to all default metrics (ex: --labels foo=bar) (default: {}, env: METRICS_LABELS)\n  --collect-default-metrics \u003cboolean\u003e  Collect default metrics (default: \"false\", env: COLLECT_DEFAULT_METRICS)\n  --headers [string...]                Set headers that will be sent on every requests (ex: --headers X-HEADER=headerA) (default: {}, env: HEADERS)\n  --final-blocks-only \u003cboolean\u003e        Only process blocks that have pass finality, to prevent any reorg and undo signal by staying further away from the chain HEAD (default: \"false\", env: FINAL_BLOCKS_ONLY)\n  --verbose \u003cboolean\u003e                  Enable verbose logging (default: \"false\", env: VERBOSE)\n  --webhook-url \u003cstring\u003e               Webhook URL to send POST (env: WEBHOOK_URL)\n  --private-key \u003cstring\u003e               Ed25519 private key to sign POST data payload (env: PRIVATE_KEY)\n  --maximum-attempts \u003cnumber\u003e          Maximum attempts to retry POST (default: 100, env: MAXIMUM_ATTEMPTS)\n  -h, --help                           display help for command\n```\n\n## Docker environment\n\nPull from GitHub Container registry\n```bash\ndocker pull ghcr.io/pinax-network/substreams-sink-webhook:latest\n```\n\nRun with `.env` file\n```bash\ndocker run -it --rm --env-file .env ghcr.io/pinax-network/substreams-sink-webhook:latest run\n```\n\nBuild from source\n```bash\ndocker build -t substreams-sink-webhook .\n```\n\n## Features\n\n- [x] POST data to URL\n- [x] Include Substreams Manifest to payload\n  - substreamsEndpoint\n  - chain\n  - finalBlockOnly\n  - moduleName\n  - type\n  - moduleHash\n- [x] Include Substreams Clock to payload\n  - timestamp\n  - number\n  - id\n- [x] Includes Substreams Session to payload\n  - traceId\n  - resolvedStartBlock\n- [x] Includes Substreams cursor to payload\n- [x] Signing policy\n  - [x] TweetNaCl\n  - [ ] ~~R1 private keys~~\n  - [ ] ~~Ether.js~~\n- [x] All messages are sent in block order, no need to parallelize\n- [x] Support for Final Blocks Only `--final-blocks-only`\n- [x] Support for Production Mode `--production-mode`\n- [x] Retry policy\n  - [x] Exponential backoff (2x)\n  - [x] Initial Interval (1s)\n  - [x] Maximum Attempts (Infinity)\n  - [x] Maximum Interval (100 * initialInterval)\n- [x] Dockerfile\n- [x] Provide CLI arguments or Environment Variables (`.env`)\n- [x] Allow params injection via `--params` or `-p`\n- [x] Prometheus metrics\n  - [x] includes metrics from `substreams-sink`\n  - [x] HTTP POST requests\n  - [x] HTTP errors\n- [x] PING URL on start (invalid + valid)\n- [x] Save `cursor.lock` only after successful POST\n  - [ ] Enforce retry policy on HTTP cursor updates\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpinax-network%2Fsubstreams-sink-webhook","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpinax-network%2Fsubstreams-sink-webhook","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpinax-network%2Fsubstreams-sink-webhook/lists"}