{"id":13651620,"url":"https://github.com/EOSIO/history-tools","last_synced_at":"2025-04-22T22:31:43.337Z","repository":{"id":38455622,"uuid":"159573050","full_name":"EOSIO/history-tools","owner":"EOSIO","description":"EOSIO History Tools","archived":true,"fork":false,"pushed_at":"2022-07-20T14:06:17.000Z","size":3020,"stargazers_count":65,"open_issues_count":46,"forks_count":45,"subscribers_count":25,"default_branch":"master","last_synced_at":"2024-01-24T04:13:24.363Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://eosio.github.io/history-tools/","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EOSIO.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-11-28T22:20:21.000Z","updated_at":"2023-02-20T22:53:41.000Z","dependencies_parsed_at":"2022-08-19T17:12:51.980Z","dependency_job_id":null,"html_url":"https://github.com/EOSIO/history-tools","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EOSIO%2Fhistory-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EOSIO%2Fhistory-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EOSIO%2Fhistory-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EOSIO%2Fhistory-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EOSIO","download_url":"https://codeload.github.com/EOSIO/history-tools/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223906290,"owners_count":17223045,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T02:00:50.984Z","updated_at":"2024-11-10T02:31:08.868Z","avatar_url":"https://github.com/EOSIO.png","language":"C++","funding_links":[],"categories":["Developers"],"sub_categories":["Libraries and Frameworks"],"readme":"# Description\n\nThis release contains the first release of EOSIO History-Tools `fill-pg` v1.0.0 and is compatible with EOSIO v2.1.0. All other tools included in the prior alpha releases of History-Tools are now deprecated.\n\n## fill-pg\n\n`fill-pg` fills postgresql with data from nodeos's State History Plugin. It provides nearly all\ndata that applications which monitor the chain need. It provides the following:\n\n* Header information from each block\n* Transaction and action traces, including inline actions and deferred transactions\n* Contract table history, at the block level\n* Tables which track the history of chain state, including\n  * Accounts, including permissions and linkauths\n  * Account resource limits and usage\n  * Contract code\n  * Contract ABIs\n  * Consensus parameters\n  * Activated consensus upgrades\n\n### PostgreSQL table schema changes\n\nThis release completely rewrites the SHiP protocol to SQL conversion code so that the database tables would directly align with the data structures defined in the SHiP protocol. This also changes the table schema used by previous releases.\n\nHere are the basic rules for the conversion:\n\n  - Nested SHiP `struct` types with more than one field are mapped to SQL custom types.\n  - Nested SHiP `vector` types are mapped to SQL arrays.\n  - SHiP `variant` types are mapped to a SQL type or table containing the union fields of their constituent types.  \n\nConsequently, instead of having their own tables in previous releases, `action_trace`, `action_trace_ram_delta`, `action_trace_auth_sequence` and `action_trace_authorization` are arrays nested inside `transaction_trace` table or `action_trace` type. The SQL `UNNEST` operator can be used to flatten arrays into tables for query. \n\nThe current list of tables created by  `fill-pg` are:\n  - account\n  - account_metadata\n  - block_info  \n  - code                      \n  - contract_index_double\n  - contract_index_long_double\n  - contract_index128\n  - contract_index256\n  - contract_index64 \n  - contract_row\n  - contract_table\n  - fill_status\n  - generated_transaction\n  - global_property\n  - key_value\n  - permission\n  - permission_link\n  - protocol_state\n  - received_block  \n  - resource_limits\n  - resource_limits_config\n  - resource_limits_state\n  - resource_usage\n  - transaction_trace \n\n## Data migration from prior releases\n\nThis release upgrades `fill-pg` to support `nodeos` v2.1.0. The remaining tools are still disabled and have not been upgraded. The schema used by\n`fill-pg` has changed. At present no data migration tool is available, so data may be manually migrated, regenerated by replaying\nfrom the desired block number, or exist side by side in two schemas, with older blocks in the v0.3.0 schema and newer blocks in the new schema.  Use of the `--pg-schema` option will facilitate the transition. \n\nFull details of the differences can be found via diff of plain text backups of each schema using `pg_dump`, not included here\nfor brevity.\n\n\n## Additional details\n\n`fill-pg` keeps action data and contract table data in its original binary form. Future versions\nmay optionally support converting this to JSON.\n\nTo conserve space, `fill-pg` doesn't store blocks in postgresql. The majority of apps\ndon't need the blocks since:\n\n* Blocks don't include inline actions and only include some deferred transactions. Most\n  applications need to handle these, so they should examine the traces instead. e.g. many transfers\n  live in the inline actions and deferred transactions that blocks exclude.\n* Most apps don't verify block signatures. If they do, then they should connect directly to\n  nodeos's State History Plugin to get the necessary data. Note that contrary to\n  popular belief, the data returned by the `/v1/get_block` RPC API is insufficient for\n  signature verification since it uses a lossy JSON conversion.\n* Most apps which currently use the `/v1/get_block` RPC API (e.g. `eosjs`) only need a tiny\n  subset of the data within each block; `fill-pg` stores this data. There are apps which use\n  `/v1/get_block` incorrectly since their authors didn't realize the blocks miss\n  critical data that their applications need.\n\n`fill-pg` supports both full history and partial history (`trim` option). This allows users\nto make their own tradeoffs. They can choose between supporting queries covering the entire\nhistory of the chain, or save space by only covering recent history.\n\n\n### SHiP protocol changes ([#10268](https://github.com/EOSIO/eos/pull/10268))\n\nSHiP protocol has been changed to allow a client to request the block_header only instead of the entire block. `fill-pg` has been updated to utilize this feature when the `nodeos` it connects to supports it. \n\n## Deprecation and Removal Notices\n`fill-rocksdb`, `wasm-ql-rocksdb`, `combo-rocksdb`,`wasm-ql-pg`, and `history-tools` have been deprecated and disabled as of this v1.0.0 release.\n\n\n## Getting Started\n\nYou can use the docker-compose file to see how fill-pg interacts with nodeos and postgresql.\nFor example, if you want to run fill-pg with some snapshot and setup one peer address, create a\n.env file such as:\n\n```\nSNAPSHOT_FILE=/root/history-tools/snapshot-2021-03-05-10-eos-v4@0171570460.bin\nPEER_ADDR=peer.main.alohaeos.com:9876\n```\n\nThen execute the command:\n\n```\ndocker-compose up\n```\n\nAnd you will start seeing logs from the 3 containers, showing how they interact between each other.\n\nFurther customization can be achieved with other environment variables for example to set an specific\nbranch/commit for nodeos or history-tools, such as:\n\n```\nDOCKER_EOSIO_TAG=develop\nDOCKER_HISTORY_TOOLS_TAG=935650a6fb9ca596affe0a3c42e6a1966675061d\n```\n\nYou can also modify the provided docker-compose.yaml so that, for example, it takes more p2p peer addresses,\nfor example:\n\n```\n       ...\n       - --p2p-peer-address=peer.main.alohaeos.com:9876\n       - --p2p-peer-address=p2p.eosflare.io:9876\n       - --p2p-peer-address=p2p.eosargentina.io:5222\n       - --p2p-peer-address=eos-bp.index.pro:9876\n       - --p2p-peer-address=eosbp-0.atticlab.net:9876\n       - --p2p-peer-address=mainnet.eosarabia.net:3571\n       ...\n```\n\n\n## Contributing\n\n[Contributing Guide](./CONTRIBUTING.md)\n\n[Code of Conduct](./CONTRIBUTING.md#conduct)\n\n## License\n\n[MIT](./LICENSE)\n\n## Important\n\nSee [LICENSE](LICENSE) for copyright and license terms.\n\nAll repositories and other materials are provided subject to the terms of this [IMPORTANT](important.md) notice and you must familiarize yourself with its terms.  The notice contains important information, limitations and restrictions relating to our software, publications, trademarks, third-party resources, and forward-looking statements.  By accessing any of our repositories and other materials, you accept and agree to the terms of the notice.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FEOSIO%2Fhistory-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FEOSIO%2Fhistory-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FEOSIO%2Fhistory-tools/lists"}