{"id":20834009,"url":"https://github.com/trueblocks/trueblocks-compare","last_synced_at":"2025-03-12T08:45:41.213Z","repository":{"id":211994774,"uuid":"730383059","full_name":"TrueBlocks/trueblocks-compare","owner":"TrueBlocks","description":"A repository used to compare different data source against TrueBlocks code","archived":false,"fork":false,"pushed_at":"2024-09-18T15:52:15.000Z","size":38742,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-18T21:30:20.262Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TrueBlocks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-11T20:03:10.000Z","updated_at":"2024-09-18T15:52:19.000Z","dependencies_parsed_at":"2024-06-10T13:00:38.502Z","dependency_job_id":"1891bc00-7860-4137-ae1f-c7aafb7e079e","html_url":"https://github.com/TrueBlocks/trueblocks-compare","commit_stats":null,"previous_names":["trueblocks/trueblocks-compare"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TrueBlocks%2Ftrueblocks-compare","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TrueBlocks%2Ftrueblocks-compare/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TrueBlocks%2Ftrueblocks-compare/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TrueBlocks%2Ftrueblocks-compare/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TrueBlocks","download_url":"https://codeload.github.com/TrueBlocks/trueblocks-compare/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243188203,"owners_count":20250453,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-18T00:17:47.791Z","updated_at":"2025-03-12T08:45:41.191Z","avatar_url":"https://github.com/TrueBlocks.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TrueBlocks Comparison with other indexers\n\nA repository used to compare other indexers against TrueBlocks.\nThe methodology and results are described in [Comparison](./results/with-3-providers-2024-06-26.md)\n\n- [TrueBlocks Comparison with other indexers](#trueblocks-comparison-with-other-indexers)\n  - [Running](#running)\n  - [Folder Structure](#folder-structure)\n  - [Code Structure](#code-structure)\n  - [The Addresses.txt File](#the-addressestxt-file)\n  - [The Code](#the-code)\n    - [Downloading the data](#downloading-the-data)\n    - [Comparing the data](#comparing-the-data)\n    - [Why does TrueBlocks find more appearances?](#why-does-trueblocks-find-more-appearances)\n  - [List of Comparisons](#list-of-comparisons)\n\n## Running\nPrepare `addresses.txt` file with addresses that should be used for comparison (`data.tar.gz` file has a list of 1,000 addresses).\nThen run:\n\n```shell\ngo run . addresses.txt\n```\n\nFor 1,000 addresses it takes couple of days to finish. Please see below for the explanation why it takes so much time (for impatient: the cause is other providers' rate limiting, which doesn't happen with local software like TrueBlocks).\n\nAfter the comparison is done the results are printed to the screen and raw data is preserved in the SQLite database (date and time is used as the file name).\nIf you'd like to only print the results again, without downloading the data, you can call:\n```shell\ngo run . --reuse path/to/database_file.sqlite\n```\n\nBy the default the results are present in human-readable, textual form. You can change to CSV (which can be pasted to a spreadsheet app for example) using `format` flag:\n```shell\ngo run . --reuse path/to/database_file.sqlite --format csv\n```\n\n## Folder Structure\n\nThis is what your directory structure should look like if you wish to reproduce the results:\n\n```[shell]\n.                      # The root of the repo. Where the code is stored.\n├── bin                # The location of the built file.\n└── store              # The location of all the data.\n    └── addresses.txt  # The list of addresses used in the comparison\n```\n\n## Code Structure\n\nThe code is written in GoLang and is located in the root of the repo. It is split into 4 files:\n\n```[shell]\n├── main.go        # The main file. It is used to run the code\n├── database.go    # The file containing the code to initialize and query SQLite database\n├── setup.go       # The file containing preparation code\n└── result.go      # The file containing the code to present the results\n```\n\nResults are obtained by querying the database. The queries can be found in `database.go` file.\n\n## The Addresses.txt File\n\nAlso at the root of the repo is a file called `addresses.txt.`. This is the list of addresses we compared. It is used to download the data from EtherScan and `chifra`. Feel free to replace this file with your own list of addresses.\n\n## The Code\n\nThe code to run the comparison is located in `main.go`. Read this very simple file.\n\n### Downloading the data\n\nThe `setup.go` file contains the code used to download the data from each source. It reads the `addresses.txt` file and processes each line using TrueBlocks SDK. The code first filters out the addresses that have too few or too many appearances. If `chifra` is installed on the machine, `chifra list` is used for filtering. If TrueBlocks Key Endpoint is configured in `trueBlocks.toml` file (path to the file can be obtained by calling `chifra config --paths`), then TrueBlocks Key is used instead. If none of the mentioned can be used, an error is returned.\n\nIf there's not too many appearances (Etherscan doesn't download more than 10,000 records, so we ignore addresses with more than 10,000 records), we procede to download from the provider and store `address, block number, transaction index, provider name` in `appearances` SQL table.\n\nIf there are too many or too few appearances, the address is saved in `incompatible_addresses` together with the number of appearances.\n\nCurrently supported providers are Alchemy, Covalent Etherscan and TrueBlocks Key. However, the code will only use providers for which API keys (or Endpoint in case of TrueBlocks Key) are defined in `trueBlocks.toml`.\n\nThe code used to download the data looks like this:\n\n```go\nopts := sdk.SlurpOptions{\n  Source: PROVIDER_ID,\n  Addrs:  []string{address},\n  Parts:  ALL_SUPPORTED,\n}\nappearances, _, err := opts.SlurpAppearances()\n```\n\nNote that the `SlurpOptions` command has a `Parts` field. It is set to every value supported by the given provider. This means it hits all eight of Etherscan's API endpoints: _normal transactions_, _external transactions_, _withdrawals_, etc., all five Alchemy's API and so on. This is the only way to get all the data from most providers. This, when combined with providers rate limiting, means that this process takes a long time to run. `chifra list` is WAY faster.\n\nThe download code runs unless you provide `--reuse path/to/existing/database_file.sqlite` flag.\n\n### Comparing the data\n\nTo ease comparing the data, a view grouping appearances and providers is present in the database:\n\n```sql\nCREATE VIEW IF NOT EXISTS view_appearances_with_providers AS SELECT\n  id,\n  address,\n  block_number,\n  transaction_index,\n  JSON_GROUP_ARRAY ( provider ) as providers\nFROM (SELECT DISTINCT * FROM appearances)\nGROUP BY address, block_number, transaction_index;\n```\n\nWe need to use `SELECT DISTINCT * FROM appearances`, because Etherscan's API endpoints return duplicates.\nAn example record stored in the view would be:\n\n```\n132|0x007b003c4d0145b512286494d5ae123aeef29d9e|4982726|173|[\"key\",\"etherscan\",\"covalent\",\"alchemy\"]\n```\nWhich can be read as: _an appearance with ID 132 of address 0x007b003c4d0145b512286494d5ae123aeef29d9e that has happened in block number 4982726, transaction index 173 was reported by all four providers_\n\nTo compare the data, different SQL queries are used. They can be found in`database.go` file.\n\nComparison has basically two possible outcomes:\n\n1. An appearance is reported by more than 1 provider\n2. An appearance is reported only by 1 provider\n\n### Why does TrueBlocks find more appearances?\n\nHopefully TrueBlocks will find more appearances than other sources. In order to check where these additional appearances come from, for each appearance we call TrueBlocks SDK `TransactionsUniq()` method. It returns `reason` - a string explaining where the appearance has been found.\n\nWe store reasons together with provider name and appearance ID in `appearance_reasons` table:\n```sql\nSELECT * FROM appearance_reasons LIMIT 1;\n-- Returns 1|key|log_923_topic_3|\n```\n\nFor non-TrueBlocks sources the reason is the API endpoint used:\n```sql\nSELECT * FROM appearance_reasons WHERE provider = 'etherscan' LIMIT 1;\n-- Returns 133|etherscan|ext|\n```\n\nWe also check if the transaction involved a balance change. We detect it by calling TrueBlocks SDK again. Please refer to `getChifraBalanceChange` function for the details.\nInformation about balance change is stored in `appearance_balance_changes` table defined as follows:\n\n```sql\nCREATE TABLE appearance_balance_changes (\n\tappearance_id INTEGER NOT NULL,\n\tbalance_change BOOLEAN,\n\tforeign key(appearance_id) references appearances(id)\n);\n```\n\n## List of Comparisons\n\nWe've written a number of comparisons with other data sources. They are listed here:\n\n| Name                                                                                                                                                             | Date       |\n| ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |\n| [TrueBlocks / Alchemy, Covalent and Etherscan Comparison](./results/with-3-providers-2024-06-26.md)                                                              | 2024-06-26 |\n| [TrueBlocks / Etherscan Comparison](./results/with-etherscan-2023-12-13.md)                                                                                      | 2023-12-13 |\n| [TrueBlocks / Covalent Comparison](https://medium.com/coinmonks/trueblocks-covalent-comparison-7b42f3d1e6f7)                                                     | 2022-09-20 |\n| [The Difference Between TrueBlocks and The Graph](https://trueblocks.io/papers/2021/the-difference-between-trueBlocks-and-rotki-and-trueBlocks-and-thegraph.pdf) | 2021-04-02 |\n| [How Accurate is Etherscan](https://tjayrush.medium.com/how-accurate-is-etherscan-83dab12eeedd)                                                                  | 2020-06-11 |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrueblocks%2Ftrueblocks-compare","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftrueblocks%2Ftrueblocks-compare","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrueblocks%2Ftrueblocks-compare/lists"}