{"id":24269810,"url":"https://github.com/pteich/elastic-query-export","last_synced_at":"2025-04-05T23:08:44.334Z","repository":{"id":42233205,"uuid":"112967051","full_name":"pteich/elastic-query-export","owner":"pteich","description":"🚚 Export Data from ElasticSearch to CSV/JSON using a Lucene Query (e.g. from Kibana) or a raw JSON Query string","archived":false,"fork":false,"pushed_at":"2025-01-28T08:38:44.000Z","size":73,"stargazers_count":92,"open_issues_count":4,"forks_count":18,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-03-29T22:11:11.651Z","etag":null,"topics":["csv","elasticsearch","export","golang","kibana"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pteich.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"pteich"}},"created_at":"2017-12-03T21:55:32.000Z","updated_at":"2025-03-19T07:31:56.000Z","dependencies_parsed_at":"2025-02-27T18:17:11.196Z","dependency_job_id":"f95a5cbb-db45-42c6-a5d6-1c059933dada","html_url":"https://github.com/pteich/elastic-query-export","commit_stats":null,"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pteich%2Felastic-query-export","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pteich%2Felastic-query-export/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pteich%2Felastic-query-export/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pteich%2Felastic-query-export/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pteich","download_url":"https://codeload.github.com/pteich/elastic-query-export/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247411234,"owners_count":20934653,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv","elasticsearch","export","golang","kibana"],"created_at":"2025-01-15T15:08:17.722Z","updated_at":"2025-04-05T23:08:44.302Z","avatar_url":"https://github.com/pteich.png","language":"Go","funding_links":["https://github.com/sponsors/pteich"],"categories":[],"sub_categories":[],"readme":"# elastic-query-export\n\nExport Data from ElasticSearch to CSV by Raw or Lucene Query (e.g. from Kibana).\nWorks with ElasticSearch 6+ (OpenSearch works too) and makes use of ElasticSearch's Scroll API and Go's\nconcurrency features to work as fast as possible.\n\n## Install\n\nDownload a pre-compiled binary for your operating system from here: https://github.com/pteich/elastic-query-export/releases\nYou need just this binary. It works on OSX (Darwin), Linux and Windows.\n\nThere are also prebuilt RPM, DEB and APK packages for your Linux distribution.\n\n### Brew\n\nUse Brew to install:\n```shell\nbrew tap pteich/tap\nbrew install elastic-query-export\n```\n\n### Arch AUR\n\n```shell\nyay -S elastic-query-export-bin\n```\n\n### Docker\n\nA Docker image is available here: https://github.com/pteich/elastic-query-export/pkgs/container/elastic-query-export\nIt can be used just like the locally installed binary: \n\n```shell\ndocker run ghcr.io/pteich/elastic-query-export:1.6.2 -h\n```\n\n\n## General usage\n\n````shell\nes-query-export -c \"http://localhost:9200\" -i \"logstash-*\" --start=\"2019-04-04T12:15:00\" --fields=\"RemoteHost,RequestTime,Timestamp,RequestUri,RequestProtocol,Agent\" -q \"RequestUri:*export*\"\n````\n\n## CLI Options\n\n| Flag             | Default               |                                                                                                         | \n|------------------|-----------------------|---------------------------------------------------------------------------------------------------------|\n| `-h --help`      |                       | show help                                                                                               |\n| `-v --version`   |                       | show version                                                                                            |\n| `-c --connect`   | http://localhost:9200 | URI to ElasticSearch instance                                                                           | \n| `-i --index`     | logs-*                | name of index to use, use globbing characters * to match multiple                                       |\n| `-q --query`     |                       | Lucene query to match documents (same as in Kibana)                                                     |\n| `   --fields`    |                       | define a comma separated list of fields to export                                                       |\n| `-o --outfile`   | output.csv            | name of output file, you can use `-` as filename to output data to stdout and pipe it to other commands |\n| `-f --outformat` | csv                   | format of the output data: possible values csv, json, raw                                               |\n| `-r --rawquery`  |                       | optional raw ElasticSearch query JSON string                                                            |\n| `-s --start`     |                       | optional start date - Format: YYYY-MM-DDThh:mm:ss.SSSZ. or any other Elasticsearch default format       |\n| `-e --end`       |                       | optional end date - Format: YYYY-MM-DDThh:mm:ss.SSSZ. or any other Elasticsearch default format         |\n| `--timefield`    |                       | optional time field to use, default to @timestamp                                                       |\n| `--verifySSL`    | false                 | optional define how to handle SSL certificates                                                          |\n| `--user`         |                       | optional username                                                                                       |\n| `--pass`         |                       | optional password                                                                                       |\n| `--size`         | 1000                  | size of the scroll window, the more the faster the export works but it adds more pressure on your nodes |\n| `--trace`        | false                 | enable trace mode to debug queries send to ElasticSearch                                                |\n\n## Output Formats\n\n- `csv` - all or selected fields separated by comma (,) with field names in the first line \n- `json` - all or selected fields as JSON objects, one per line\n- `raw` - JSON dump of matching documents including id, index and _source field containing the document data. One document as JSON object per line.\n\n## Pipe output to other commands\n\nSince v1.6.0 you can provide `-` as filename and send output to stdout. This can be used to pipe it to other commands like so:\n\n```shell\nes-query-export -start=\"2019-04-04T12:15:00\" -q \"RequestUri:*export*\" -outfile - | aws s3 cp - s3://mybucket/stream.csv\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpteich%2Felastic-query-export","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpteich%2Felastic-query-export","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpteich%2Felastic-query-export/lists"}