{"id":13491842,"url":"https://github.com/skratchdot/elasticsearch-tools","last_synced_at":"2025-04-06T14:12:04.896Z","repository":{"id":19779892,"uuid":"23038607","full_name":"skratchdot/elasticsearch-tools","owner":"skratchdot","description":"A collection of elasticsearch command line tools for doing things like bulk importing/exporting and exporting/importing mappings.","archived":false,"fork":false,"pushed_at":"2023-01-03T22:04:10.000Z","size":844,"stargazers_count":190,"open_issues_count":27,"forks_count":44,"subscribers_count":18,"default_branch":"master","last_synced_at":"2025-03-30T11:07:04.521Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/skratchdot.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-MIT","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-08-17T10:57:16.000Z","updated_at":"2024-07-12T10:29:36.000Z","dependencies_parsed_at":"2023-01-13T20:34:59.327Z","dependency_job_id":null,"html_url":"https://github.com/skratchdot/elasticsearch-tools","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skratchdot%2Felasticsearch-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skratchdot%2Felasticsearch-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skratchdot%2Felasticsearch-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skratchdot%2Felasticsearch-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/skratchdot","download_url":"https://codeload.github.com/skratchdot/elasticsearch-tools/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247492557,"owners_count":20947545,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T19:01:00.732Z","updated_at":"2025-04-06T14:12:04.855Z","avatar_url":"https://github.com/skratchdot.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"# elasticsearch-tools\n\nA collection of elasticsearch command line tools for doing things like bulk importing/exporting\nand exporting/importing mappings.\n\nIt was created because some of the existing import/export tools ran too slow on my machine. Using\nthe new bulk API seemed to speed things up dramatically. The other tools I used also weren't\nexporting \\_parent and \\_routing fields.\n\n## Installation\n\n```bash\nnpm install -g elasticsearch-tools\n```\n\nAfter installing, you will have access to the following command line tools:\n\n#### Exporting\n\n- [es-export-bulk](#usage-es-export-bulk)\n- [es-export-mappings](#usage-es-export-mappings)\n- [es-export-settings](#usage-es-export-settings)\n- [es-export-aliases](#usage-es-export-aliases)\n\n#### Importing\n\n- [es-import-bulk](#usage-es-import-bulk)\n- [es-import-mappings](#usage-es-import-mappings)\n- [es-import-settings](#usage-es-import-settings)\n- [es-import-aliases](#usage-es-import-aliases)\n\n## Usage: es-export-bulk\n\n### Options\n\n```bash\nes-export-bulk --help\n\nUsage: es-export-bulk [options]\n\n  Options:\n\n    -h, --help                           output usage information\n    -v, --version                        output the version number\n    -u, --url \u003curl\u003e                      comma-separated elasticsearch urls to connect to\n    -f, --file \u003cfile\u003e                    the file to write data to\n    -m, --max \u003cnumber\u003e                   the maximum number of items to export. different than the scroll size\n    --transformMeta \u003cjs\u003e                 a javascript function that returns an object that is the transformed meta object\n    --transformSource \u003cjs\u003e               a javascript function that returns an object that is the transformed source object\n    --transformMetaInit \u003cjs\u003e             a javascript function that returns an init object that contains helpers for the transform function\n    --transformSourceInit \u003cjs\u003e           a javascript function that returns an init object that contains helpers for the transform function\n    --index \u003cindex\u003e                      ES OPTION: a comma-separated list of index names to search; use _all or empty string to perform the operation on all indices\n    --type \u003ctype\u003e                        ES OPTION: a comma-separated list of document types to search; leave empty to perform the operation on all types\n    --body \u003cbody\u003e                        ES OPTION: the body to send along with this request.\n    --analyzer \u003canalyzer\u003e                ES OPTION: The analyzer to use for the query string\n    --analyzeWildcard \u003canalyzeWildcard\u003e  ES OPTION: specify whether wildcard and prefix queries should be analyzed (default: false)\n    --fields \u003cfields\u003e                    ES OPTION: a comma-separated list of fields to return as part of a hit (default: \"*\")\n    --from \u003cfrom\u003e                        ES OPTION: starting offset (default: 0)\n    --q \u003cq\u003e                              ES OPTION: query in the Lucene query string syntax\n    --routing \u003crouting\u003e                  ES OPTION: a comma-separated list of specific routing values\n    --scroll \u003cscroll\u003e                    ES OPTION: specify how long a consistent view of the index should be maintained for scrolled search (default: 1m)\n    --size \u003csize\u003e                        ES OPTION: number of hits to return during each scan\n    --sort \u003csort\u003e                        ES OPTION: a comma-separated list of \u003cfield\u003e:\u003cdirection\u003e pairs\n    --timeout \u003ctimeout\u003e                  ES OPTION: explicit operation timeout\n    --apiVersion \u003capiVersion\u003e            ES CLIENT OPTION: the major version of the Elasticsearch nodes you will be connecting to (default: 2.3)\n    --maxRetries \u003cmaxRetries\u003e            ES CLIENT OPTION: how many times should the client try to connect to other nodes before returning a ConnectionFault error (default: 3)\n    --requestTimeout \u003crequestTimeout\u003e    ES CLIENT OPTION: milliseconds before an HTTP request will be aborted and retried. This can also be set per request (default: 30000)\n    --deadTimeout \u003cdeadTimeout\u003e          ES CLIENT OPTION: milliseconds that a dead connection will wait before attempting to revive itself (default: 60000)\n    --pingTimeout \u003cpingTimeout\u003e          ES CLIENT OPTION: milliseconds that a ping request can take before timing out (default: 3000)\n    --maxSockets \u003cmaxSockets\u003e            ES CLIENT OPTION: maximum number of concurrent requests that can be made to any node (default: 10)\n    --minSockets \u003cminSockets\u003e            ES CLIENT OPTION: minimum number of sockets to keep connected to a node (default: 10)\n    --selector \u003cselector\u003e                ES CLIENT OPTION: select a connection from the ConnectionPool using roundRobin (default) or random\n```\n\n### Examples\n\n#### export 1 hour of data from local db\n\n```bash\nes-export-bulk --url http://localhost:9200 --file ~/backups/elasticsearch/prod/data.json --body '\n{\"query\":{\"range\":{\"timestamp\":{\"gte\":\"2014-08-13T11:00:00.000Z\",\"lte\":\"2014-08-13T12:00:00.000Z\"}}}}\n'\n```\n\n#### export \"myIndex\" from local db\n\n```bash\nes-export-bulk --url http://localhost:9200 --file ~/backups/elasticsearch/prod/data.json --index myIndex\n```\n\n#### add a key/value to all exported documents\n\n```bash\nes-export-bulk --url http://localhost:9200 --file ~/backups/elasticsearch/prod/data.json --transformSource 'data.foo = \"neat\"'\n# the return statement is optional\nes-export-bulk --url http://localhost:9200 --file ~/backups/elasticsearch/prod/data.json --transformSource 'data.foo = \"neat\";return data;'\n```\n\n#### delete the key \"foo\" from all exported documents\n\n```bash\nes-export-bulk --url http://localhost:9200 --file ~/backups/elasticsearch/prod/data.json --transformSource 'delete data.foo'\n```\n\n#### don't include \\_parent in meta data\n\n```bash\nes-export-bulk --url http://localhost:9200 --file ~/backups/elasticsearch/prod/data.json --transformMeta 'delete data.index._parent'\n```\n\n#### change the index name that we export\n\n```bash\nes-export-bulk --url http://localhost:9200 --file ~/backups/elasticsearch/prod/data.json --transformMeta 'data.index._index = \"newIndex\"'\n```\n\n## Usage: es-export-mappings\n\n### Options\n\n```bash\nes-export-mappings --help\n\nUsage: es-export-mappings [options]\n\n  Options:\n\n    -h, --help                               output usage information\n    -v, --version                            output the version number\n    -u, --url \u003curl\u003e                          the elasticsearch url to connect to\n    -f, --file \u003cfile\u003e                        the file to write data to\n    --index \u003cindex\u003e                          ES OPTION: String, String[], Boolean — A comma-separated list of index names\n    --type \u003ctype\u003e                            ES OPTION: String, String[], Boolean — A comma-separated list of document types\n    --ignoreUnavailable \u003cignoreUnavailable\u003e  ES OPTION: Boolean — Whether specified concrete indices should be ignored when unavailable (missing or closed)\n    --allowNoIndices \u003callowNoIndices\u003e        ES OPTION: Boolean — Whether to ignore if a wildcard indices expression resolves into no concrete indices. (This includes _all string or when no indices have been specified)\n    --expandWildcards \u003cexpandWildcards\u003e      ES OPTION: String — Whether to expand wildcard expression to concrete indices that are open, closed or both.\n    --local \u003clocal\u003e                          ES OPTION: Boolean — Return local information, do not retrieve the state from master node (default: false)\n```\n\n### Examples\n\n#### export mappings from local db\n\n```bash\nes-export-mappings --url http://localhost:9200 --file ~/backups/elasticsearch/prod/prod.mappings.json\n```\n\n## Usage: es-export-settings\n\n### Options\n\n```bash\nes-export-settings --help\n\nUsage: es-export-settings [options]\n\n  Options:\n\n    -h, --help                               output usage information\n    -v, --version                            output the version number\n    -u, --url \u003curl\u003e                          the elasticsearch url to connect to\n    -f, --file \u003cfile\u003e                        the file to write data to\n    --index \u003cindex\u003e                          ES OPTION: String, String[], Boolean — A comma-separated list of index names\n    --ignoreUnavailable \u003cignoreUnavailable\u003e  ES OPTION: Boolean — Whether specified concrete indices should be ignored when unavailable (missing or closed)\n    --allowNoIndices \u003callowNoIndices\u003e        ES OPTION: Boolean — Whether to ignore if a wildcard indices expression resolves into no concrete indices. (This includes _all string or when no indices have been specified)\n    --expandWildcards \u003cexpandWildcards\u003e      ES OPTION: String — Whether to expand wildcard expression to concrete indices that are open, closed or both.\n    --local \u003clocal\u003e                          ES OPTION: Boolean — Return local information, do not retrieve the state from master node (default: false)\n    --name \u003cname\u003e                            ES OPTION: String, String[], Boolean — The name of the settings that should be included\n```\n\n### Examples\n\n#### export settings from local db\n\n```bash\nes-export-settings --url http://localhost:9200 --file ~/backups/elasticsearch/prod/prod.settings.json\n```\n\n## Usage: es-export-aliases\n\n### Options\n\n```bash\nes-export-aliases --help\n\nUsage: es-export-aliases [options]\n\n  Options:\n\n    -h, --help         output usage information\n    -v, --version      output the version number\n    -u, --url \u003curl\u003e    the elasticsearch url to connect to\n    -f, --file \u003cfile\u003e  the file to write data to\n    --index \u003cindex\u003e    ES OPTION: String, String[], Boolean — A comma-separated list of index names\n    --local \u003clocal\u003e    ES OPTION: Boolean — Return local information, do not retrieve the state from master node (default: false)\n    --name \u003cname\u003e      ES OPTION: String, String[], Boolean — The name of the settings that should be included\n```\n\n### Examples\n\n#### export aliases from local db\n\n```bash\nes-export-aliases --url http://localhost:9200 --file ~/backups/elasticsearch/prod/prod.aliases.json\n```\n\n## Usage: es-import-bulk\n\n### Options\n\n```bash\nes-import-bulk --help\n\nUsage: es-import-bulk [options]\n\n  Options:\n\n    -v, --version          output the version number\n    -u, --url \u003curl\u003e        the elasticsearch url to connect to\n    -f, --file \u003cfile\u003e      the file to read data from\n    -m, --max \u003citems\u003e      the max number of lines to process per batch (default: 20,000) (default: 20000)\n    --requestTimeout \u003cms\u003e  ES CLIENT OPTION: milliseconds before an HTTP request will be aborted and retried. This can also be set per request (default: 30000) (default: 30000)\n    -h, --help             output usage information\n```\n\n### Examples\n\n#### import data to local db from file\n\n```bash\nes-import-bulk --url http://localhost:9200 --file ~/backups/elasticsearch/prod/rafflev1.json\n```\n\n## Usage: es-import-mappings\n\n### Options\n\n```bash\nes-import-mappings --help\n\nUsage: es-import-mappings [options]\n\n  Options:\n\n    -v, --version                            output the version number\n    -u, --url \u003curl\u003e                          the elasticsearch url to connect to\n    -f, --file \u003cfile\u003e                        the file to read data from\n    --ignoreConflicts \u003cignoreConflicts\u003e      ES OPTION: Boolean — Specify whether to ignore conflicts while updating the mapping (default: false)\n    --timeout \u003ctimeout\u003e                      ES OPTION: Date, Number — Explicit operation timeout\n    --masterTimeout \u003cmasterTimeout\u003e          ES OPTION: Date, Number — Specify timeout for connection to master\n    --ignoreUnavailable \u003cignoreUnavailable\u003e  ES OPTION: Boolean — Whether specified concrete indices should be ignored when unavailable (missing or closed)\n    --allowNoIndices \u003callowNoIndices\u003e        ES OPTION: Boolean — Whether to ignore if a wildcard indices expression resolves into no concrete indices. (This includes _all string or when no indices have been specified)\n    --expandWildcards \u003cexpandWildcards\u003e      ES OPTION: String — Whether to expand wildcard expression to concrete indices that are open, closed or both.\n    -h, --help                               output usage information\n```\n\n### Examples\n\n#### import mappings to local db\n\n```bash\nes-import-mappings --url http://localhost:9200 --file ~/backups/elasticsearch/prod/prod.mappings.json\n```\n\n## Usage: es-import-settings\n\n### Options\n\n```bash\nes-import-settings --help\n\nUsage: es-import-settings [options]\n\n  Options:\n\n    -v, --version                            output the version number\n    -u, --url \u003curl\u003e                          the elasticsearch url to connect to\n    -f, --file \u003cfile\u003e                        the file to read data from\n    --masterTimeout \u003cmasterTimeout\u003e          ES OPTION: Date, Number — Specify timeout for connection to master\n    --ignoreUnavailable \u003cignoreUnavailable\u003e  ES OPTION: Boolean — Whether specified concrete indices should be ignored when unavailable (missing or closed)\n    --allowNoIndices \u003callowNoIndices\u003e        ES OPTION: Boolean — Whether to ignore if a wildcard indices expression resolves into no concrete indices. (This includes _all string or when no indices have been specified)\n    --expandWildcards \u003cexpandWildcards\u003e      ES OPTION: String — Whether to expand wildcard expression to concrete indices that are open, closed or both.\n    -h, --help                               output usage information\n```\n\n### Examples\n\n#### import settings to local db\n\n```bash\nes-import-settings --url http://localhost:9200 --file ~/backups/elasticsearch/prod/prod.settings.json\n```\n\n## Usage: es-import-aliases\n\n### Options\n\n```bash\nes-import-aliases --help\n\nUsage: es-import-aliases [options]\n\n  Options:\n\n    -v, --version                    output the version number\n    -u, --url \u003curl\u003e                  the elasticsearch url to connect to\n    -f, --file \u003cfile\u003e                the file to read data from\n    --timeout \u003ctimeout\u003e              ES OPTION: Date, Number — Explicit operation timeout\n    --masterTimeout \u003cmasterTimeout\u003e  ES OPTION: Date, Number — Specify timeout for connection to master\n    -h, --help                       output usage information\n```\n\n### Examples\n\n#### import aliases to local db\n\n```bash\nes-import-aliases --url http://localhost:9200 --file ~/backups/elasticsearch/prod/prod.aliases.json\n```\n\n## Other Elasticsearch Tools\n\n#### Imports / Exports\n\n- [elasticdump](https://github.com/taskrabbit/elasticsearch-dump)\n- [elasticsearch-exporter](https://github.com/mallocator/Elasticsearch-Exporter)\n\n## Running tests\n\nUnit tests can be ran via:\n\n```bash\nnpm run test\n```\n\nThe integration tests hit an elasticsearch server at: `localhost:20202`. To\nstart the server, you can install [docker](https://www.docker.com), then run:\n\n```bash\ndocker-compose up\n```\n\nOne the server is running, you can run the integration tests via:\n\n```bash\nnpm run test:integration\n```\n\n## License\n\nCopyright (c) 2014 skratchdot\nLicensed under the MIT license.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskratchdot%2Felasticsearch-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fskratchdot%2Felasticsearch-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskratchdot%2Felasticsearch-tools/lists"}