{"id":27958990,"url":"https://github.com/yxuco/ethdecoder","last_synced_at":"2026-04-24T16:03:19.752Z","repository":{"id":96633954,"uuid":"419111720","full_name":"yxuco/ethdecoder","owner":"yxuco","description":"This CLI decodes Ethereum transactions and events, stores results in CouchDB, and then exports customized views to CSV files for data visualization and analysis.","archived":false,"fork":false,"pushed_at":"2021-11-07T02:45:48.000Z","size":419,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-07T18:37:10.657Z","etag":null,"topics":["data-analysis","decoding","ethereum"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yxuco.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-10-19T22:40:14.000Z","updated_at":"2021-11-07T02:45:51.000Z","dependencies_parsed_at":null,"dependency_job_id":"1d7fedbb-8d1e-4656-9317-37266b750a25","html_url":"https://github.com/yxuco/ethdecoder","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/yxuco/ethdecoder","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yxuco%2Fethdecoder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yxuco%2Fethdecoder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yxuco%2Fethdecoder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yxuco%2Fethdecoder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yxuco","download_url":"https://codeload.github.com/yxuco/ethdecoder/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yxuco%2Fethdecoder/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32230421,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-24T13:21:15.438Z","status":"ssl_error","status_checked_at":"2026-04-24T13:21:15.005Z","response_time":64,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-analysis","decoding","ethereum"],"created_at":"2025-05-07T18:27:19.769Z","updated_at":"2026-04-24T16:03:19.747Z","avatar_url":"https://github.com/yxuco.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Ethereum Data Decoder\n\nThis [Node.js](https://nodejs.org/) application retrieves Ethereum data from [BigQuery](https://console.cloud.google.com/bigquery) public dataset, decodes the transactions and events, and then writes the results in a [CouchDB](https://docs.couchdb.org/) database.\n\n## Prerequisite\n\n### Install Node.js\n\nOn my Mac, I used [nvm](https://github.com/nvm-sh/nvm#installing-and-updating) to install Node.js as follows:\n\n```bash\ntouch ~/.zshrc\ncurl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash\nnvm install node\n```\n\n### Create BigQuery Project\n\nSign in to Google Platform and create a BigQuery project via [link](https://console.cloud.google.com/flows/enableapi?apiid=bigquery.googleapis.com).\n\nCreate a service account and download the account key as described [here](https://cloud.google.com/docs/authentication/getting-started).  Store the downloaded file in the [config](./config) folder.  Edit the [config.json](./config.json) to set `bqCredential` to this key file.\n\n### Create Etherscan API Key\n\nSign in to [Etherscan](https://etherscan.io/), and create an API Key via [link](https://etherscan.io/myapikey).  Edit the [config.json](./config.json) to set `etherscanApiKey` to this key.  We call Etherscan API to fetch contract ABIs.\n\n### Install CouchDB\n\nFollow the [instruction](https://docs.couchdb.org/en/latest/install/index.html) to install Apache CouchDB.  View the databases by using the `Fauxton UI` at [http://127.0.0.1:5984/_utils/](http://127.0.0.1:5984/_utils/).\n\nCreate a user and a database used to store decoded Ethereum transactions and events by using the following script, e.g.,\n\n```bash\ncurl -X PUT -u admin:password http://127.0.0.1:5984/ethdb\ncurl -X PUT -u admin:password http://localhost:5984/_users/org.couchdb.user:ethadmin -H \"Accept: application/json\" -H \"Content-Type: application/json\" -d '{\"name\": \"ethadmin\", \"password\": \"ethadmin\", \"roles\": [\"ethdb_admin\"], \"type\": \"user\"}'\ncurl -X PUT -u admin:password http://localhost:5984/_users/org.couchdb.user:ethuser -H \"Accept: application/json\" -H \"Content-Type: application/json\" -d '{\"name\": \"ethuser\", \"password\": \"ethuser\", \"roles\": [\"ethdb_member\"], \"type\": \"user\"}'\ncurl -X PUT -u admin:password http://localhost:5984/ethdb/_security -H \"Content-Type: application/json\" -d '{\"admins\": { \"names\": [], \"roles\": [\"_admin\", \"ethdb_admin\"] }, \"members\": { \"names\": [], \"roles\": [\"_admin\", \"ethdb_admin\", \"ethdb_member\"] } }'\n```\n\nThis example creates a database `ethdb` and a user `ethuser` for storing the application output.  The database connection info must be configured in [config.json](./config.json).\n\n## Configuration\n\nBesides system connection data as described in the previous section, the [config.json](./config.json) can provide the following attributes to customize the behavior of the decoder:\n\n* `standardAbis` is a list of standard token ABI files, e.g., [erc20.json](./abis/standard/erc20.json) or [erc721.json](./abis/standard/erc721.json), etc, which are used to decode transaction inputs and event data if the required contract ABI cannot be fetched from Etherscan.\n* `tokenInfo` is a file that lists metadata of token contracts that are not available in on-chain blocks.  The metadata specifies the `symbol`, `name`, and `decimals` of token contracts.\n* `contractAbis` is a folder that contains ABI files for contracts that does not provide verified ABI on the Etherscan.  The ABI files in this folder must be named after the corresponding contract address, e.g., [0x6b175474e89094c44da98b954eedeac495271d0f.json](./abis/0x6b175474e89094c44da98b954eedeac495271d0f.json) is the ABI file for the [DAI token](https://etherscan.io/address/0x6b175474e89094c44da98b954eedeac495271d0f#code).\n\nBefore start collecting Ethereum data, you must create at least the `contract` and `transaction` views in the CouchDB as described in [views/README.md](./views/README.md).\n\n## Run\n\nThe command line interface supports the following functions:\n\n* `decode` command fetches Ethereum transactions and events from BigQuery public data store, decodes them and then writes the result to a CouchDB database.  It also builds a reference cache of contract ABIs by using Etherscan API calls.\n* `update` command updates the contract cache in batches by fetching metadata and related token info from BigQuery public data store.\n* `export` command queries a CouchDB view, and format the result as a CSV file, so it can be imported to a data analytics tool.\n\n### Decode\n\nRun the following command to decode Ethereum transactions and events from a specified contract during a specified date range:\n\n```bash\nnode index.js decode contract-address [ start-date [ end-date ]]\n```\n\nThe `contract-address` may be a comma-delimited list of contract addresses.\n\nFor example, the following command would decode the data for the `DAI` token contract in the date range from `2021-10-01` and `2021-10-05` (inclusive):\n\n```bash\nnode index.js decode '0x6b175474e89094c44da98b954eedeac495271d0f' '2021-10-01' '2021-10-03'\n```\n\nIf the `start-date` and/or `end-date` is not specified, the default date would be the date before the system date, i.e., yesterday.\n\n### Update\n\nRun the following command to update cached contracts created during the decode process:\n\n```bash\nnode index.js update\n```\n\n### Export\n\nThe following command fetches data from a CouchDB view by specified aggregation level:\n\n```bash\nnode index.js export ddoc view params [output [options]]\n```\n\nwhere `ddoc` and `view` specifies the name of the CouchDB design document and the name of the view defined in the `ddoc`; `params` specifies the query parameters as defined by the [CouchDB API](https://docs.couchdb.org/en/stable/api/ddoc/views.html#db-design-design-doc-view-view-name); `output` is the name of the output report file.  The result will be printed on `stdout` if the file name is not specified.\n\nBy default, the reported token counts will be in fractional units defined by the `decimals` of corresponding tokens, e.g., `USDC` is in unit of 10\u003csup\u003e-6\u003c/sup\u003e, while `DAI` is in unit of 10\u003csup\u003e-18\u003c/sup\u003e.  If you want to report tokens with normalized counts, you can use the parameter `options` to specify the token key column corresponding to the reported value of a token count.  For example, `options = {value: 0}` would mean that the first key column is the token address that will be used to normalize the `value` column that contains the corresponding token count.\n\nIf you have collected some Uniswap v2 data, and defined the view in [uniswap-v2.json](./views/uniswap-v2.json), you can execute a query on the view by using the following command:\n\n```bash\nnode index.js export 'uniswap-v2' 'swap-token-out' '{\"group_level\": 5, \"limit\": 20}' './report.csv'\n```\n\nIt will generate a comma-delimited file `report.csv` containing the first 20 rows of the aggregated view.  This view reports transactions that swaps pairs of tokens, including swapped amount of tokens as `amountIn` and `amountOut`.  To report only tokens with known token decimals, you can specify the options as:\n\n```bash\nnode index.js export 'uniswap-v2' 'swap-token-out' '{\"group_level\": 5, \"limit\": 200}' './report.csv' '{\"amountIn\": 0, \"amountOut\": 1}'\n```\n\nIn this report, the input token count `amountIn` will be normalized by using the token address in the first key column, and the output token count `amountOut` will be normalized by using the token address in the second key column.\n\nThe `options` argument may also specify one or more value filters to reduce the size of the resulting output file.  The filters specify functions that evaluate attributes of the view's value object and return true for exported rows.  For example, following option will export only values with `count \u003e 2`:\n\n```json\n{ \"$filter\": \"x =\u003e x.count \u003e 2\" }\n```\n\nSee [example.sh](./reports/example.sh) for a sample command that uses options for both amount normalization and value filters.\n\nYou can also specify key-filters to filter exported rows by keys of a view, e.g.,\n\n```json\n{ \"$keyFilter\": \"x =\u003e x[0] === '0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48'\" }\n```\n\nThe above key filter would export only rows whows first key element is the specified contract address.  If you want to specify multiple filters, you can specify `$filter` and/or `$keyFilter` as an array of strings, and each string defines a filter function for the view.\n\nNote that you may preview the result of a query by using the CouchDB Fauxton UI before you execute the `export` command.\n\n## Data in CouchDB\n\nThe decoded data are stored in the CouchDB as JSON documents.  Each document is tagged with a `docType` of `contract`, `transaction`, or `event`, depending on the corresponding content type.\n\nBy creating CouchDB views, the collected data of transactions and events can be indexed and aggregated by map and reduce on CouchDB server.  Some sample views are described in [views](./views).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyxuco%2Fethdecoder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyxuco%2Fethdecoder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyxuco%2Fethdecoder/lists"}