{"id":20272346,"url":"https://github.com/steemit/hive2elastic","last_synced_at":"2025-07-21T10:02:37.613Z","repository":{"id":42963378,"uuid":"193753685","full_name":"steemit/hive2elastic","owner":"steemit","description":"Fork of https://github.com/esteemapp/hive2elastic","archived":false,"fork":false,"pushed_at":"2024-08-09T01:54:47.000Z","size":53,"stargazers_count":1,"open_issues_count":1,"forks_count":6,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-11T04:42:02.635Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/steemit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-06-25T17:29:26.000Z","updated_at":"2022-03-25T21:00:56.000Z","dependencies_parsed_at":"2025-04-11T04:42:05.976Z","dependency_job_id":"df1b15ef-744e-4284-a030-e6ab76765231","html_url":"https://github.com/steemit/hive2elastic","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/steemit/hive2elastic","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/steemit%2Fhive2elastic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/steemit%2Fhive2elastic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/steemit%2Fhive2elastic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/steemit%2Fhive2elastic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/steemit","download_url":"https://codeload.github.com/steemit/hive2elastic/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/steemit%2Fhive2elastic/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266278125,"owners_count":23904036,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T12:43:03.775Z","updated_at":"2025-07-21T10:02:37.589Z","avatar_url":"https://github.com/steemit.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# hive2elastic\n\nhive2elastic synchronises [hive](https://github.com/steemit/hivemind)'s hive_posts_cache table to a elasticsearch index and keeps it updated.\n\n\n## Before start\n\nSome additional database objects have to be created on hive's database.\n\n**Follow steps below:**\n\n1- Stop hive. Make sure all hive processes stopped.\n\n2- Create database objects on hive's database.\n\n```\nCREATE TABLE __h2e_posts\n(\n    post_id INTEGER PRIMARY KEY\n);\nCREATE TABLE __h2e_accounts\n(\n    account_id INTEGER PRIMARY KEY\n);\n```\n\n```\nINSERT INTO __h2e_posts (post_id) SELECT post_id FROM hive_posts_cache;\nINSERT INTO __h2e_accounts (account_id) SELECT id FROM hive_accounts;\n```\n\n```\nCREATE OR REPLACE FUNCTION __fn_h2e_posts()\n  RETURNS TRIGGER AS\n$func$\nBEGIN   \n    IF NOT EXISTS (SELECT post_id FROM __h2e_posts WHERE post_id = NEW.post_id) THEN\n    \tINSERT INTO __h2e_posts (post_id) VALUES (NEW.post_id);\n\tEND IF;\n\tRETURN NEW;\nEND\n$func$ LANGUAGE plpgsql;\n\nCREATE OR REPLACE FUNCTION __fn_h2e_accounts()\n  RETURNS TRIGGER AS\n$func$\nBEGIN   \n    IF NOT EXISTS (SELECT account_id FROM __h2e_accounts WHERE account_id = NEW.id) THEN\n    \tINSERT INTO __h2e_accounts (account_id) VALUES (NEW.id);\n\tEND IF;\n\tRETURN NEW;\nEND\n$func$ LANGUAGE plpgsql;\n```\n\n```\nCREATE TRIGGER __trg_h2e_posts\nAFTER INSERT OR UPDATE ON hive_posts_cache\nFOR EACH ROW EXECUTE PROCEDURE __fn_h2e_posts();\n\nCREATE TRIGGER __trg_h2e_accounts\nAFTER INSERT OR UPDATE ON hive_accounts\nFOR EACH ROW EXECUTE PROCEDURE __fn_h2e_accounts();\n```\n\n3- Start hive\n\n*Make sure database credentials that you use has delete permission on __h2e_posts and __h2e_accounts table*\n\n**Elasticsearch** \n\nYou can find detailed installation instructions [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html)\n\n\n## Installation\n\n```\n$ git clone https://github.com/esteemapp/hive2elastic\n$ cd hive2elastic\n$ pip3 install -e .\n```\n\n## Configuration\n\nYou can configure hive2elastic by these arguments/environment variables:\n\n\n|\tArgument\t|\tEnvironment Variable\t|\tDescription | Default|\n|\t--------\t|\t--------\t|\t--------\t|  --------\t|  \n|\t--db-url\t|\tDB_URL\t|\tConnection string for hive database\t| -- | \n|\t--es-url\t|\tES_URL\t|\tElasticsearch server address\t| -- | \n|\t--es-index\t|\tES_INDEX\t|\t Index name on elasticsearch\t| hive_posts | \n|\t--es-type\t|\tES_TYPE\t|\t Type name on elasticsearch index | posts | \n|\t--es-index-reply\t|\tES_INDEX_REPLY\t|\tReply index name on elasticsearch\t| hive_replies | \n|\t--es-type-reply\t|\tES_TYPE_REPLY\t|   Reply type name on elasticsearch index | replies | \n|\t--bulk-size\t|\tBULK_SIZE\t|\t Number of documents to index in a single loop | 500 | \n|\t--max-workers\t|\tMAX_WORKERS\t|  Max workers for document preparation process | 2 | \n\n\n## Example configuration and running\n\n```\n$ export DB_URL=postgresql://username:passwd@localhost:5432/hive \n$ export ES_URL=http://localhost:9200/\n$ export BULK_SIZE=2000                 \n$ export MAX_WORKERS=4\n\n$ hive2elastic_post\n$ hive2elastic_account\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsteemit%2Fhive2elastic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsteemit%2Fhive2elastic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsteemit%2Fhive2elastic/lists"}