{"id":23695898,"url":"https://github.com/zenodeapp/donor-drop-backend","last_synced_at":"2025-09-03T00:31:46.500Z","repository":{"id":269218206,"uuid":"906755377","full_name":"zenodeapp/donor-drop-backend","owner":"zenodeapp","description":null,"archived":false,"fork":false,"pushed_at":"2024-12-23T01:22:32.000Z","size":15,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-12-30T05:57:45.246Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zenodeapp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-21T20:08:12.000Z","updated_at":"2024-12-30T02:00:53.000Z","dependencies_parsed_at":"2024-12-21T21:22:11.064Z","dependency_job_id":"4011414e-e306-44d0-bdf4-f592854fe53b","html_url":"https://github.com/zenodeapp/donor-drop-backend","commit_stats":null,"previous_names":["zenodeapp/donation-drop-backend","zenodeapp/donor-drop-backend"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zenodeapp%2Fdonor-drop-backend","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zenodeapp%2Fdonor-drop-backend/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zenodeapp%2Fdonor-drop-backend/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zenodeapp%2Fdonor-drop-backend/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zenodeapp","download_url":"https://codeload.github.com/zenodeapp/donor-drop-backend/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":231819022,"owners_count":18431234,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-30T05:57:47.959Z","updated_at":"2025-09-03T00:31:46.459Z","avatar_url":"https://github.com/zenodeapp.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Donor Drop Backend\n\nThis was initially written by chimmykk and bengtlofgren. It has been optimized, bug-fixed, and enhanced with additional logic to meet the requirements of the donor drop campaign and the official frontend. This project is licensed under the MIT-license (see [LICENSE](./LICENSE)).\n\n# Overview\n\n- Contains SQL database which can be setup via Docker\n- Contains scraper which is dependent on the existence of this SQL database.\n\n## Installation\n\n### 1. Install dependencies\n```\nnpm install\n```\n\nOR\n\n```\nyarn install\n```\n\n### 2. .env configuration\nCreate an .env file in the root. See [.env.example](.env.example) for an example.\n\n### 3. Setup postgres\n\nBefore continuing, read the comments in the [docker-compose.yml](./docker-compose.yml) file and configure this properly.\n\n```bash\ndocker-compose up -d\n```\n\nThis will setup the correct postgres database running on `POSTGRES_PORT` (default port: 5434). The tables and views that will get created are specified in the [init-scripts/init.sql](./init-scripts/init.sql) file.\n\nFor easy access to the database, you could use a tool like `pgAdmin` or `dbeaver` using the credentials specified in your .env file.\n\n\u003e [!NOTE]\n\u003e\n\u003e For a quick reset, use the script [`clean-start.sh`](./clean-start.sh).\n\u003e \n\u003e **IMPORTANT:** this will wipe the ENTIRE database.\n\n\n### 4. Run scraper\n\n\u003e [!TIP]\n\u003e\n\u003e Use a separate systemctl service to run the scraper. See issue [#22](https://github.com/zenodeapp/donor-drop-backend/issues/22) for a template. \n\n```bash\nnode scraper.mjs\n```\n\n#### 4.1. Scraper options _(optional)_\n\nThere are currently two flags one could run the scraper with:\n\n- `--once`\n\n  ```\n  node scraper.mjs --once\n  ```\n\n  This will only let the scraper do a single run. Useful if you just want to fetch data once, without letting it check Etherscan/Infura every n-seconds.\n\n\n- `--all-etherscan-txs`\n\n  ```\n  node scraper.mjs --all-etherscan-txs\n  ```\n  \n  \u003e This flag will act as if `--once` is set as well.\n\n  This will get all transactions that meet the conditions described in [A.1 Donation finality](#a1-donation-finality) **without doing any memo or tnam validation**. This is useful if we want to give the people that made a mistake during the donor drop the opportunity to link their tnams again. See [A.2 Rescue plan](#a2-rescue-plan) for a detailed description on how to approach that.\n\n## Results\n\nThe results are built from three different categories:\n\n- (1) List of _eligible users_\n- (2) List of users who donated _after the cap got reached_\n- (3) List of users who _initially weren't included due to mistakes made_, but got corrected using [A.2 Rescue plan](#a2-rescue-plan)\n\nThere are two ways one could export these results. Either [_by using the wizard_](#i-using-the-wizard-recommended) or [_by using PSQL_](#ii-using-psql).\n\n### I. Using the wizard _(recommended)_\n\n#### Command\nThe following command can be used to export the raw and final (merged) results in .csv, .json and .proposal.json-format:\n\n```\nnode result.mjs\n```\n\n#### Arguments _(optional)_\n\n- `--exclude-eligibles`: excludes eligible users\n- `--exclude-above-cap`: excludes users who donated after the cap\n- `--exclude-not-in-db`: excludes users who were initially not included\n- `--min-eth-per-address x`: the minimum amount of ETH a participant is required to have donated\n  - _default value_: `0.03`\n- `--max-eth-per-address y`: the maximum amount of ETH a participant was allowed to donate\n  - _default value_: `0.3`\n- `--clean`: this will clean the ./output folder before exporting the results\n\n#### Features\n- Exports .csv, .json and .proposal.json files.\n- Saves _raw_ exports from the views:\n  - _private_result_eligible_addresses_finalized_in_db_ as  `_raw_eligibles` (1),\n  - _private_result_above_cap_addresses_in_db_ as `_raw_above_cap` (2) and\n  - _private_result_addresses_not_in_db_ as `_raw_not_in_db` (3).\n- Merges the results together by grouping the results by _tnam address_. This makes sure each participant will only be included once in the .proposal.json file.\n- Gives the user a heads up whenever a participant gets skipped due to a _missing tnam address_ or _signature hash_.\n- Asks the user whether to _cap a tnam address' ETH amount_ if the resulting value _is higher than_ the set `--max-eth-per-address`.\n- Asks the user whether to _exclude a tnam address_ if this participant's resulting donation amount _is lower than_ the set `--min-eth-per-address`.\n  \n### II. Using PSQL\nThis route requires more manual work as it will only get you the _raw_ .csv files (which are also exported when _using the wizard_). If for some reason you're unable to use `node` or you only have direct access to the SQL database, use these commands:\n- _private_result_eligible_addresses_finalized_in_db_ as  `_raw_eligibles.csv` (1)\n\n  ```sql\n  copy(SELECT from_address AS eth_address, tnam AS nam_address, eligible_amount AS eth_amount, predicted_suggested_nam AS nam_amount FROM private_result_eligible_addresses_finalized_in_db) To '/var/lib/postgresql/_raw_eligibles.csv' With CSV DELIMITER ',' HEADER;\n  ```\n\n- _private_result_above_cap_addresses_in_db_ as `_raw_above_cap.csv` (2)\n\n  ```sql\n  copy(SELECT from_address AS eth_address, tnam AS nam_address, eligible_above_cap AS eth_amount, suggested_nam AS nam_amount FROM private_result_above_cap_addresses_in_db) To '/var/lib/postgresql/_raw_above_cap.csv' With CSV DELIMITER ',' HEADER;\n  ```\n\n- _private_result_addresses_not_in_db_ as `_raw_not_in_db.csv` (3)\n\n  ```sql\n  copy(SELECT from_address AS eth_address, tnam AS nam_address, total_eth AS eth_amount, suggested_nam AS nam_amount, sig_hash FROM private_result_addresses_not_in_db) To '/var/lib/postgresql/_raw_not_in_db.csv' With CSV DELIMITER ',' HEADER;\n  ```\n\n## Appendix\n\n### A.1 Donation finality\n\nBy default the scraper will periodically check for transactions made to the address defined in your .env-file. It uses a combination of info gathered from the Etherscan and Infura API and only picks up the transactions that meet the following conditions:\n\n- donation _x_ comes from block _n_, where _n_ \u003e= `SCRAPER_START_BLOCK`.\n- donation _x_ has transaction date _d_, where _d_ \u003e= `SCRAPER_START_DATE` and _d_ \u003c= `SCRAPER_END_DATE`. \n- donation _x_ has hex _h_ in the transaction's memo-field, where decode(_h_) = _a valid tnam-address_. The decode-method is quite robust and auto-corrects most of the common mistakes people make (e.g. multi-encoded hex string, forgetting the '0x' part, adding more characters than necessary).\n- donation _x_ is not a failed transaction.\n\nThe scraper starts two schedulers: one that registers any transaction that passes the requirements above and the other that also considers block finality. \n\n\u003e [!NOTE]\n\u003e \n\u003e **Why _two schedulers_?**\n\u003e \n\u003e A transaction is only certain once a block is completely finalized on-chain. This takes on average 15 to 20 minutes. Which is problematic if we want to show a tally in real-time. So, to solve this, we temporarily use the data from the scheduler that's unbothered by finalization _as an indication_, whereas the actual results get calculated using the data from the _finalized_-scheduler. The frontend makes sure to take both this real-time and finalized data into account and visualize them accordingly.\n\n### A.2 Rescue plan\n\nIf there are people who messed up their donation, the following can be done:\n\n1. Wait for approx. 30 minutes after the donor drop ended (to make sure all eth blocks are in a `finalized` state).\n2. _(Optional)_ adjust the `SCRAPER_START_DATE`, `SCRAPER_END_DATE` and `SCRAPER_START_BLOCK` in your [.env](./.env.example)-file.\n3. Run ```node scraper.mjs --all-etherscan-txs```.\n4. Double-check the data this command gathered in the `etherscan_transactions_all`-table. It should contain every transaction done between `SCRAPER_START_DATE` and `SCRAPER_END_DATE`.\n5. Switch the frontend to the [`with-link`](https://github.com/zenodeapp/donor-drop-frontend/tree/with-link)-branch and re-deploy it.\n6. Let people link their tnam addresses using the frontend (this form will only allow wallet addresses that failed to register a tnam address).\n7. Keep track of the results by checking `unaccounted_addresses` and `private_result_addresses_not_in_db`.\n\n### A.3 Testing _(likely outdated)_\n\nThe testing suite works as follows:\n\n(If you have not already done so, please run `npm install` and set up docker-compose as described above)\n\nThe tests will be done against the database specified in the `.env` file. Ideally this would be done against a `.env.test` file, but for the purposes of this project, the `.env` file will be used.\n\nTo run the tests, use the following command:\n\n```bash\nnpm test -- --detectOpenHandles --verbose\n```\n\nAll tests should pass.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzenodeapp%2Fdonor-drop-backend","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzenodeapp%2Fdonor-drop-backend","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzenodeapp%2Fdonor-drop-backend/lists"}