{"id":1990722,"url":"https://github.com/EOSLaoMao/elasticsearch_plugin","last_synced_at":"2025-03-13T17:34:03.614Z","repository":{"id":97223109,"uuid":"147660295","full_name":"EOSLaoMao/elasticsearch_plugin","owner":"EOSLaoMao","description":"Nodeos plugin for archiving blockchain data into Elasticsearch.","archived":false,"fork":false,"pushed_at":"2019-08-10T12:49:19.000Z","size":123,"stargazers_count":57,"open_issues_count":1,"forks_count":24,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-07-31T16:33:30.974Z","etag":null,"topics":["elastic","elasticsearch-plugin","eos","nodeos-plugin"],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EOSLaoMao.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2018-09-06T10:57:05.000Z","updated_at":"2022-04-06T01:39:24.000Z","dependencies_parsed_at":"2024-01-20T15:54:45.109Z","dependency_job_id":"b1983664-58f9-4ae8-b1d6-22795daf50f3","html_url":"https://github.com/EOSLaoMao/elasticsearch_plugin","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EOSLaoMao%2Felasticsearch_plugin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EOSLaoMao%2Felasticsearch_plugin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EOSLaoMao%2Felasticsearch_plugin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EOSLaoMao%2Felasticsearch_plugin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EOSLaoMao","download_url":"https://codeload.github.com/EOSLaoMao/elasticsearch_plugin/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221391517,"owners_count":16811437,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elastic","elasticsearch-plugin","eos","nodeos-plugin"],"created_at":"2024-01-20T15:38:24.562Z","updated_at":"2024-10-25T05:31:01.095Z","avatar_url":"https://github.com/EOSLaoMao.png","language":"C++","funding_links":[],"categories":["DB Plugins:"],"sub_categories":[],"readme":"# elasticsearch_plugin\n\nNodeos plugin for archiving blockchain data into Elasticsearch, inspired by [mongo_db_plugin](https://github.com/EOSIO/eos/tree/master/plugins/mongo_db_plugin).\n\n**Currently the plugin only work with [official eosio repository](https://github.com/EOSIO/eos).**\n\n## Getting Help\n- For questions about this project:\n  - [Join our Telegram group](https://t.me/eosesplugin)\n  - [Open an issue](https://github.com/EOSLaoMao/elasticsearch_plugin/issues/new)\n\n## Indices\n\nIt is recommended to use other tools for indices management. Checkout [EOSLaoMao/elasticsearch-node](https://github.com/EOSLaoMao/elasticsearch-node).  \n\n[Document examples](#document-examples)\n\n## Benchmark\n\nDetail: [Benchmark](./benchmark/benchmark.md)\n\n### Replay 10000 Block\n\n|                      | elapse(s) | speed(b/s) |\n| -------------------- |:---------:|:----------:|\n| elasticsearch_plugin | 266       | 37.59      |\n| mongo_db_plugin      | 694       | 14.41      |\n\n### Replay 100000 Block\n\n|                      | elapse(s) | speed(b/s) |\n| -------------------- |:---------:|:----------:|\n| elasticsearch_plugin | 354       | 282.49     |\n| mongo_db_plugin      | 987       | 101.32     |\n\n## Performance Tuning\n\nExample filters:\n\n```text\n--elastic-filter-out=eosio:onblock:\n--elastic-filter-out=gu2tembqgage::\n--elastic-filter-out=blocktwitter::\n```\n\n[Tune for indexing speed](https://www.elastic.co/guide/en/elasticsearch/reference/master/tune-for-indexing-speed.html)\n\nIn the benchmark, `elasticsearch_plugin` is running with default config. For production deploy, you can tweak some config.\n\n```text\n  --elastic-thread-pool-size arg (=4)            The size of the data processing thread pool.\n  --elastic-bulk-size-mb arg (=5)                The size(megabytes) of the each bulk request.\n```\n\n## Installation\n\n### Install `EOSLaoMao/elasticlient`\n\n`elasticsearh_plugin` rely on [EOSLaoMao/elasticlient](https://github.com/EOSLaoMao/elasticlient)\n\n```bash\ngit clone https://github.com/WLBF/elasticlient.git\ncd elasticlient\ngit submodule update --init --recursive\ncmake -DBUILD_ELASTICLIENT_TESTS=NO -DBUILD_ELASTICLIENT_EXAMPLE=NO\nmake\nsudo make install\n# copy cpr library manually\nsudo cp -r \"external/cpr/include/cpr\" \"/usr/local/include/cpr\"\nsudo cp \"lib/libcpr.so\" \"/usr/local/lib/libcpr.so\"\n```\n\n### Embed `elasticsearch_plugin` into `nodeos`\n\n1. Get `elasticsearch_plugin` source code.\n\n```bash\ngit clone https://github.com/EOSLaoMao/elasticsearch_plugin.git plugins/elasticsearch_plugin\ncd plugins/elasticsearch_plugin\ngit submodule update --init --recursive\n```\n\n2. Add subdirectory to `plugins/CMakeLists.txt`.\n\n```cmake\n...\nadd_subdirectory(mongo_db_plugin)\nadd_subdirectory(login_plugin)\nadd_subdirectory(login_plugin)\nadd_subdirectory(elasticsearch_plugin) # add this line.\n...\n```\n\n3. Add following line to `programs/nodeos/CMakeLists.txt`.\n\n```cmake\ntarget_link_libraries( ${NODE_EXECUTABLE_NAME}\n        PRIVATE appbase\n        PRIVATE -Wl,${whole_archive_flag} login_plugin               -Wl,${no_whole_archive_flag}\n        PRIVATE -Wl,${whole_archive_flag} history_plugin             -Wl,${no_whole_archive_flag}\n        ...\n        # add this line.\n        PRIVATE -Wl,${whole_archive_flag} elasticsearch_plugin       -Wl,${no_whole_archive_flag}\n        ...\n```\n\n## Usage\n\nThe usage of `elasticsearch_plugin` is similar to [mongo_db_plugin](https://github.com/EOSIO/eos/tree/master/plugins/mongo_db_plugin). It is recommended that a large `--abi-serializer-max-time-ms` value be passed into the nodeos running the elasticsearch_plugin as the default abi serializer time limit is not large enough to serialize large blocks.\n\n```plain\nConfig Options for eosio::elasticsearch_plugin.\n  -q [ --elastic-queue-size ] arg (=1024)                       The target queue size between nodeos \n                                                                and elasticsearch plugin thread.\n  --elastic-thread-pool-size arg (=4)                           The size of the data processing thread \n                                                                pool.\n  --elastic-bulk-size-mb arg (=5)                               The size(megabytes) of the each bulk \n                                                                request.\n  --elastic-abi-db-size-mb arg (=1024)                          Maximum size(megabytes) of the abi \n                                                                database.\n  --elastic-block-start arg (=0)                                If specified then only abi data pushed \n                                                                to elasticsearch until specified block \n                                                                is reached.\n  -u [ --elastic-url ] arg                                      elasticsearch URL connection string If \n                                                                not specified then plugin is disabled.\n  --elastic-user arg                                            elasticsearch user.\n  --elastic-password arg                                        elasticsearch password.\n  --elastic-store-blocks arg (=1)                               Enables storing blocks in \n                                                                elasticsearch.\n  --elastic-store-block-states arg (=1)                         Enables storing block state in \n                                                                elasticsearch.\n  --elastic-store-transactions arg (=1)                         Enables storing transactions in \n                                                                elasticsearch.\n  --elastic-store-transaction-traces arg (=1)                   Enables storing transaction traces in \n                                                                elasticsearch.\n  --elastic-store-action-traces arg (=1)                        Enables storing action traces in \n                                                                elasticsearch.\n  --elastic-filter-on arg                                       Track actions which match \n                                                                receiver:action:actor. Receiver, \n                                                                Action, \u0026 Actor may be blank to include\n                                                                all. i.e. eosio:: or :transfer:  Use * \n                                                                or leave unspecified to include all.\n  --elastic-filter-out arg                                      Do not track actions which match \n                                                                receiver:action:actor. Receiver, \n                                                                Action, \u0026 Actor may be blank to exclude\n                                                                all.\n  --elastic-index-accounts arg (=accounts)                      elasticsearch accounts index name.\n  --elastic-index-blocks arg (=blocks)                          elasticsearch blocks index name.\n  --elastic-index-transactions arg (=transactions)              elasticsearch transactions index name.\n  --elastic-index-block-states arg (=block_states)              elasticsearch block_states index name.\n  --elastic-index-transaction-traces arg (=transaction_traces)  elasticsearch transaction_traces index  name.\n  --elastic-index-action-traces arg (=action_traces)           elasticsearch action_traces index name.\n\n```\n\n## TODO\n\n- [ ] Due to `libcurl` [100-continue feature](https://curl.haxx.se/mail/lib-2017-07/0013.html), consider replace [EOSLaoMao/elasticlient](https://github.com/EOSLaoMao/elasticlient) with other simple http client like [https://cpp-netlib.org/#](https://cpp-netlib.org/#)\n\n\n## Document examples\n\n* accounts\n```\n{\n  \"creator\": \"eosio\",\n  \"pub_keys\": [\n    {\n      \"permission\": \"owner\",\n      \"key\": \"EOS5Ga8VeykSY7SXJyHbnanSPHPcQ3LmKDtJjJBJHokgYDxeokP4R\"\n    },\n    {\n      \"permission\": \"active\",\n      \"key\": \"EOS5Ga8VeykSY7SXJyHbnanSPHPcQ3LmKDtJjJBJHokgYDxeokP4R\"\n    }\n  ],\n  \"account_create_time\": \"2018-06-09T12:01:56.500\",\n  \"account_controls\": [],\n  \"name\": \"heztcnjtguge\"\n}\n```\n\n* action_traces\n```\n{\n  \"receipt\": {\n    \"receiver\": \"eosio.token\",\n    \"act_digest\": \"94753d3277e8759aaa7e4ee8f19cfa36013180e0d66d62a4a2196d76786d574a\",\n    \"global_sequence\": 1730634,\n    \"recv_sequence\": 459727,\n    \"auth_sequence\": [\n      [\n        \"eosio\",\n        1730627\n      ]\n    ],\n    \"code_sequence\": 1,\n    \"abi_sequence\": 1\n  },\n  \"act\": {\n    \"account\": \"eosio.token\",\n    \"name\": \"transfer\",\n    \"authorization\": [\n      {\n        \"actor\": \"eosio\",\n        \"permission\": \"active\"\n      }\n    ],\n    \"data\": \"{\\\"from\\\":\\\"eosio\\\",\\\"to\\\":\\\"eosio.ram\\\",\\\"quantity\\\":\\\"0.1219 EOS\\\",\\\"memo\\\":\\\"buy ram\\\"}\",\n    \"hex_data\": \"0000000000ea3055000090e602ea3055c30400000000000004454f5300000000076275792072616d\"\n  },\n  \"context_free\": false,\n  \"elapsed\": 80,\n  \"console\": \"\",\n  \"trx_id\": \"98a9839544b7f4ae6ebc5fa1f9f17bd31027c4d845b7af9a6fa8cc8f9fafb0f1\",\n  \"block_num\": 6643,\n  \"block_time\": \"2018-06-09T12:53:42.500\",\n  \"producer_block_id\": \"000019f31e63bace5345f703d5140daeada8453abec7d5ebeb2d735b859e8e07\",\n  \"account_ram_deltas\": [],\n  \"except\": null\n}\n```\n\n* transaction_traces\n```\n{\n  \"id\": \"711c6dab0e1b1a90aff0c54c3aed64f870a77ad35fd0aa6a19c6542c55653fd3\",\n  \"block_num\": 7167,\n  \"block_time\": \"2018-06-09T12:58:06.000\",\n  \"producer_block_id\": \"00001bff0cc43235d99514029440013e2fb7b6bd5eb6fb5c848954a088a9e0f9\",\n  \"receipt\": {\n    \"status\": \"executed\",\n    \"cpu_usage_us\": 129052,\n    \"net_usage_words\": 1298\n  },\n  \"elapsed\": 44456,\n  \"net_usage\": 10384,\n  \"scheduled\": false,\n  \"action_traces\": [\n    {\n      \"receipt\": {\n        \"receiver\": \"eosio\",\n        \"act_digest\": \"6556d890f6238c244020411322d6a40b5c03519e3da17dc05f525387ae850914\",\n        \"global_sequence\": 1874256,\n        \"recv_sequence\": 878486,\n        \"auth_sequence\": [\n          [\n            \"eosio\",\n            1874249\n          ]\n        ],\n        \"code_sequence\": 2,\n        \"abi_sequence\": 2\n      },\n      \"act\": {\n        \"account\": \"eosio\",\n        \"name\": \"newaccount\",\n        \"authorization\": [\n          {\n            \"actor\": \"eosio\",\n            \"permission\": \"active\"\n          }\n        ],\n        \"data\": {\n          \"creator\": \"eosio\",\n          \"name\": \"gi4tcnzwhege\",\n          \"owner\": {\n            \"threshold\": 1,\n            \"keys\": [\n              {\n                \"key\": \"EOS8M7Qbuq2U5PBph9QhBm5o7sHzuxCgMewwWnmLk9be2WshZEXZB\",\n                \"weight\": 1\n              }\n            ],\n            \"accounts\": [],\n            \"waits\": []\n          },\n          \"active\": {\n            \"threshold\": 1,\n            \"keys\": [\n              {\n                \"key\": \"EOS8M7Qbuq2U5PBph9QhBm5o7sHzuxCgMewwWnmLk9be2WshZEXZB\",\n                \"weight\": 1\n              }\n            ],\n            \"accounts\": [],\n            \"waits\": []\n          }\n        },\n        \"hex_data\": \"0000000000ea3055a0986afc4f94896301000000010003c7893323d9b3bd2ad4aab6708177e17a7254fdd28cc5700a54fbe3d5869e424d0100000001000000010003c7893323d9b3bd2ad4aab6708177e17a7254fdd28cc5700a54fbe3d5869e424d01000000\"\n      },\n      \"context_free\": false,\n      \"elapsed\": 184,\n      \"console\": \"\",\n      \"trx_id\": \"711c6dab0e1b1a90aff0c54c3aed64f870a77ad35fd0aa6a19c6542c55653fd3\",\n      \"block_num\": 7167,\n      \"block_time\": \"2018-06-09T12:58:06.000\",\n      \"producer_block_id\": \"00001bff0cc43235d99514029440013e2fb7b6bd5eb6fb5c848954a088a9e0f9\",\n      \"account_ram_deltas\": [\n        {\n          \"account\": \"gi4tcnzwhege\",\n          \"delta\": 2996\n        }\n      ],\n      \"except\": null,\n      \"inline_traces\": []\n    }\n  ],\n  \"except\": null\n}\n```\n\n* blocks\n```\n{\n  \"timestamp\": \"2018-06-09T12:03:37.500\",\n  \"producer\": \"eosio\",\n  \"confirmed\": 0,\n  \"previous\": \"00000288461e6ef331cb87ebf3e8fda122dc4c1b5cfd313b080a762ce0a472df\",\n  \"transaction_mroot\": \"a850a72dc0fc67adce701455551c1d82246fae12055f5d10b66ba55921a0dc34\",\n  \"action_mroot\": \"1b79fd3dd0e544c4eacce998cf6d9a7fa9a3d4533590779b94d2be002fd52618\",\n  \"schedule_version\": 0,\n  \"new_producers\": null,\n  \"header_extensions\": [],\n  \"producer_signature\": \"SIG_K1_KfX4VW3aLd2KjsJCqWQpDdauT7xJwERXpckJhUG8UPNANtgqVjRZEJBGKH7DJ7ZJr9E1mmTkzqSz4KQ2FyLBpAv4KPqpxt\",\n  \"transactions\": [...],\n  \"block_extensions\": [],\n  \"irreversible\": true,\n  \"validated\": true\n}\n```\n\n* block_states\n```\n{\n  \"active_schedule\": {\n    \"version\": 1,\n    \"producers\": [\n      {\n        \"producer_name\": \"genesisblock\",\n        \"block_signing_key\": \"EOS8Yid3mE5bwWMvGGKYEDxFRGHostu5xCzFanyJP1UdgZ5mpPdwZ\"\n      }\n    ]\n  },\n  \"in_current_chain\": true,\n  \"pending_schedule_lib_num\": 12149,\n  \"dpos_proposed_irreversible_blocknum\": 154026,\n  \"dpos_irreversible_blocknum\": 154025,\n  \"pending_schedule\": {\n    \"version\": 1,\n    \"producers\": []\n  },\n  \"producer_to_last_produced\": [\n    [\n      \"eosio\",\n      12150\n    ],\n    [\n      \"genesisblock\",\n      154026\n    ]\n  ],\n  \"pending_schedule_hash\": \"c43882d5411af19d8596d5d835b3f4bd6a7fd36cc4c7fb55942ef11f8d1473b6\",\n  \"blockroot_merkle\": {\n    \"_active_nodes\": [\n      \"000259a9acccbad8e577c2ec8bd87e2e956b21ebdf1524c01ea65dd464ce0fec\",\n      \"3323314d8d9f2eed9233911c01ec52a0b22ec59c5b2902ac9396080bad602a90\",\n      \"df39e691954565362fbedd6be43667b551594a7e37ad1f7e2d4f8f25c545b1e8\",\n      \"6931158aae9e5f5391ca559eb0503f1631de3ddff73785d3bf98a12418cbfe77\",\n      \"e4e2fdd9850efba91ed0f12efcecebe54b37ce9264b50722948dd23a8fb49f62\",\n      \"2930747c325841aa6926c3b8143402bb5e35c88f1a5af76cf2c9c09761be9863\",\n      \"2cde046c52b281ed55abb64a9b3da2882c19bd8100734bf9774f94b539870479\",\n      \"93eade5dac8991d23381f380f9121568c47b9e3c37317a5de3c28503aae93e53\",\n      \"827270b90af501d41051054f541ec063dfbab4e4a2eb4d6f85ffac575df777f0\",\n      \"0f8be62807a52c66552ac3ffb9406fe9b644d4c7a38fc195205c6a16c549aa3b\"\n    ],\n    \"_node_count\": 154025\n  },\n  \"confirm_count\": [],\n  \"confirmations\": [],\n  \"block_num\": 154026,\n  \"validated\": true,\n  \"bft_irreversible_blocknum\": 0,\n  \"header\": {\n    \"header_extensions\": [],\n    \"previous\": \"000259a9acccbad8e577c2ec8bd87e2e956b21ebdf1524c01ea65dd464ce0fec\",\n    \"schedule_version\": 1,\n    \"producer\": \"genesisblock\",\n    \"transaction_mroot\": \"0000000000000000000000000000000000000000000000000000000000000000\",\n    \"producer_signature\": \"SIG_K1_K7EFKTaKJrEeQLXSdmBoC15yva6otnavPq4LGNZcojuSYtsnLuPtZDBf7Sqgw3U2cGWjnS7Ncg5cY2KLNiUFMKYgAP9D3m\",\n    \"confirmed\": 0,\n    \"action_mroot\": \"18343eeab718c318d906b20cee90c8c6eeeffef04cdd514f0a22e75f24c21cf1\",\n    \"timestamp\": \"2018-06-11T08:21:42.000\"\n  },\n  \"producer_to_last_implied_irb\": [\n    [\n      \"genesisblock\",\n      154025\n    ]\n  ],\n  \"block_signing_key\": \"EOS8Yid3mE5bwWMvGGKYEDxFRGHostu5xCzFanyJP1UdgZ5mpPdwZ\",\n  \"id\": \"000259aaeecdcdd5af5196321bc74643169962a4c20dff0cacb6225c4a7ab2c2\",\n  \"irreversible\": true\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FEOSLaoMao%2Felasticsearch_plugin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FEOSLaoMao%2Felasticsearch_plugin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FEOSLaoMao%2Felasticsearch_plugin/lists"}