{"id":23545539,"url":"https://github.com/bh2smith/web3-event-etl","last_synced_at":"2025-05-15T10:34:38.092Z","repository":{"id":93410254,"uuid":"591272253","full_name":"bh2smith/web3-event-etl","owner":"bh2smith","description":"Prototype for Web3 Event Triggering ETL process resulting in DB insertion","archived":false,"fork":false,"pushed_at":"2023-01-29T21:38:51.000Z","size":166,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-17T10:47:58.637Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bh2smith.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-20T10:54:15.000Z","updated_at":"2023-01-24T15:39:40.000Z","dependencies_parsed_at":"2023-03-18T23:20:13.453Z","dependency_job_id":null,"html_url":"https://github.com/bh2smith/web3-event-etl","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bh2smith%2Fweb3-event-etl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bh2smith%2Fweb3-event-etl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bh2smith%2Fweb3-event-etl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bh2smith%2Fweb3-event-etl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bh2smith","download_url":"https://codeload.github.com/bh2smith/web3-event-etl/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254323183,"owners_count":22051739,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-26T08:18:47.190Z","updated_at":"2025-05-15T10:34:38.050Z","avatar_url":"https://github.com/bh2smith.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Web3 Event ETL\n\nFollowing this diagram\n\n![Diagram](./static/GrandScheme.png)\n\n1. Some Web3 Event occurs (`SettlementEvent`)\n2. An independent/external service (such as [Tenderly Web3 Actions](https://tenderly.co/web3-actions)) triggers script\n   defined by `docker/Dockerfile.script` passing the relevant arguments.\n3. Script Processes Event Data and Inserts into DB.\n\nNote that:\n\n- script image will have ENV vars \"baked\" in (API keys \u0026 DB credentials). So this image would have to be private.\n\n## Build \u0026 Run\n\n### Database\n\n```shell\n# Build\ndocker build -f docker/Dockerfile.db -t db-image .\n# Run\ndocker run -d --network=host db-image\n```\n\n### Script Container\n\n```shell\n# Build\ndocker build -f ./docker/Dockerfile.script -t script-image .\n# Run (requires DB running on localhost)!\ndocker run -d --network=host script-image --tx-hash 0x1\n```\n\nAlternatively, you can use the bash script:\n\n```shell\n./process_tx.sh 0x5\n```\n\nYou can now run the script-image as many times as you like with whatever string you'd like to insert.\n\n## Triggering Script Image\n\nThe database and script from above would exist in our AWS cluster as a database \u0026 AWS lambda function respectively.\n\nAccording to [this documentation](https://docs.aws.amazon.com/lambda/latest/dg/lambda-invocation.html) it is possible to\ninvoke a lambda function via POST request.\nThis would imply that the above services are independent of the options explorer made below\n\nTenderly Web3 Actions provides a serverless Smart Contract event listener from which one can execute actions defined in\nTypescript. In particular, from within the web3 action we could trigger an AWS lambda\nvia [function URL](https://docs.aws.amazon.com/lambda/latest/dg/lambda-urls.html). In particular, see\n\n- [creating and managing function URLs](https://docs.aws.amazon.com/lambda/latest/dg/urls-configuration.html)\n- [invoking function URLs](https://docs.aws.amazon.com/lambda/latest/dg/urls-invocation.html)\n\n## WebHooks\n\n### Alchemy\n\nDoesn't work. See `alchemy` directory\n\n### Tenderly\n\nWorks wonders. Version found in `tenderly` deployed at\nhttps://dashboard.tenderly.co/bh2smith/project/action/6e0a113b-39e3-44cb-b5b2-3b4e8c7db365\n\nParses Settlement Transfers and currently logs meaningful parsed data.\n\nNearly all methods are tested.\n\n#### Testing Trigger\n\n```shell\ncd tenderly/actions \u0026\u0026 npm install\nyarn test\n```\n\n#### Deployment\n\n```shell\ncd tenderly\ntenderly login\ntenderly actions deploy\n```\n\n# Deployment Ideas\n\nIn kubernetes we might have the following setup:\n\n- [Postgres Instance](https://gist.github.com/anaisbetts/2244d6517dc2cc09b4470e6f68c2bec1)\n- [Optional] [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/lambda-invocation.html) for ETL Process\n- [Webhook](https://dashboard.tenderly.co/bh2smith/project/action/6e0a113b-39e3-44cb-b5b2-3b4e8c7db365) invoking\n  the Lambda or writing directly to the database.\n\nWith all of the above setup, one would then deploy a [dune-sync](https://github.com/cowprotocol/dune-sync) cronjob which\nreads the DB and uploads to Dunes AWS bucket\n\n### Connect to AWS DB\n\nGenerate types\n\n```shell\nnpx @databases/pg-schema-cli --database $DATABASE_URL --directory src/__generated__\n```\nwhere `$DATABASE_URL` takes the form `postgresql://{user}:{password}@{host}:{port}/{database}`","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbh2smith%2Fweb3-event-etl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbh2smith%2Fweb3-event-etl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbh2smith%2Fweb3-event-etl/lists"}