{"id":26621357,"url":"https://github.com/vl4deee11/ch2es","last_synced_at":"2025-04-10T10:30:19.781Z","repository":{"id":48149125,"uuid":"373921548","full_name":"vl4deee11/ch2es","owner":"vl4deee11","description":"Application for transferring data from clickhouse to elasticsearch","archived":false,"fork":false,"pushed_at":"2021-07-14T17:45:12.000Z","size":54,"stargazers_count":4,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-24T09:14:57.352Z","etag":null,"topics":["clickhouse","database","elasticsearch","transfer"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vl4deee11.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-06-04T17:55:02.000Z","updated_at":"2021-07-14T17:37:44.000Z","dependencies_parsed_at":"2022-09-02T06:43:05.888Z","dependency_job_id":null,"html_url":"https://github.com/vl4deee11/ch2es","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vl4deee11%2Fch2es","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vl4deee11%2Fch2es/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vl4deee11%2Fch2es/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vl4deee11%2Fch2es/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vl4deee11","download_url":"https://codeload.github.com/vl4deee11/ch2es/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248199136,"owners_count":21063641,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clickhouse","database","elasticsearch","transfer"],"created_at":"2025-03-24T09:15:02.124Z","updated_at":"2025-04-10T10:30:19.766Z","avatar_url":"https://github.com/vl4deee11.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ch2es\n\nData transfer from clickhouse to elasticsearch.\n\nClickhouse reader provide 4 types of cursor (set with `--ch-cursor`)\n\n    0. offset\n        transfer data with limit and offset (use for small tables)\n        \n    1. timestamp\n        transfer data with timestamp offset (unix-timestamp \u003c= \u003c--ch-tsc-field\u003e \u003c= unix-timestamp + \u003c--ch-tsc-step\u003e).\n        start at \u003c--ch-tsc-min\u003e end at \u003c--ch-tsc-max\u003e\n        \n    2. json file\n        transfer data from file with clickhouse format JSONEachRow. line by line (use for big tables)\n        \n    3. stdin\n        transfer data from stdin with clickhouse format JSONEachRow. line by line (use for big tables)\n\nElasticsearch writer provide 2 type of converter (set with `--es-converter`)\n    \n    0. null\n        write data with unchanged clickhouse schema\n        \n    1. nested\n        write data with nested schema\n        \n# elasticsearch additional info\n1. At this stage of the project, all dots in the field name are rewritten to the value of the `--es-dot-replacer` flag, \nif it is set, otherwise the points remain in the fields, which can cause an elasticsearch error.\n\n2. In elasticsearch by default use _id filed. You can rewrite this field with `--es-id-field` flag.\n\n# the nested schema additional info \n\nSupport [nested fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html). \n\nExample:\n\nGet from clickhouse :\n```json\n{\n  \"foo\": 1,\n  \"bar\": \"string\",\n  \"baz\": [1,2,3,4],\n  \"baz2\": [\"one\", \"two\"]\n}\n```\nInsert to elasticsearch:\n```json\n{\n  \"foo\": 1,\n  \"bar\": \"string\",\n  \"\u003c--es-nc-field\u003e\" : [\n    {\n      \"baz\": 1,\n      \"baz2\": \"one\"\n    },\n    {\n      \"baz\": 2,\n      \"baz2\": \"two\"\n    },\n    {\n      \"baz\": 3,\n      \"baz2\": null // added if --es-nc-null=true\n    },\n    {\n      \"baz\": 4,\n      \"baz2\": null // added if --es-nc-null=true\n    },\n  ]\n}\n``` \n\n\n# client params\n\nUse `-h` for the client params\n\n# examples \n\n1. transfer from file\n\n```bash\n./ch2es.bin --es-bulksz=1000 --es-host=host --es-port=9201 --es-idx=idx --tn=16 --ch-cursor=2 --ch-jfc-file=./q.json --es-dot-replacer=_\n```\n\n2. transfer with timestamp cursor\n\n```bash\n./ch2es.bin --ch-fields=user_id,author_id,my_timestamp --ch-pass=xyz --ch-db=my_db --ch-host=host --ch-table=my_table --ch-query-timeout=60 --ch-conn-timeout=10 --es-bulksz=5000 --es-host=host --es-idx=my_index --tn=4 --ch-cursor=1 --ch-tsc-step=10 --ch-tsc-min=1622505600 --ch-tsc-max=1624924800 --ch-tsc-field=my_timestamp --es-dot-replacer=_\n```\n \n3. transfer with offset/limit cursor\n\n```bash\n./ch2es.bin --ch-fields=user_id,author_id,my_timestamp --ch-pass=xyz --ch-db=my_db --ch-host=host --ch-table=my_table --ch-query-timeout=60 --ch-conn-timeout=10 --es-bulksz=5000 --es-host=host --es-idx=my_index --tn=4 --ch-cursor=1 --ch-ofc-limit=10 --ch-ofc-max-offset=2000000 --ch-ofc-offset=20 --ch-ofc-order=user_id --es-dot-replacer=_\n```\n\n4. transfer from stdin\n\n```bash\nclickhouse-client -h 0.0.0.0 -q \"select * from info limit 1000 format JSONEachRow\" | ./ch2es.bin --es-bulksz=1000 --es-host=host --es-port=9200 --es-idx=test_idx --tn=1 --ch-cursor=3\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvl4deee11%2Fch2es","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvl4deee11%2Fch2es","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvl4deee11%2Fch2es/lists"}