{"id":13487753,"url":"https://github.com/jprante/elasticsearch-knapsack","last_synced_at":"2025-03-27T23:31:36.209Z","repository":{"id":5857768,"uuid":"7074625","full_name":"jprante/elasticsearch-knapsack","owner":"jprante","description":"Knapsack plugin is an import/export tool for Elasticsearch","archived":false,"fork":false,"pushed_at":"2021-11-18T00:46:41.000Z","size":7650,"stargazers_count":472,"open_issues_count":53,"forks_count":77,"subscribers_count":48,"default_branch":"master","last_synced_at":"2024-10-30T23:35:38.341Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jprante.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-12-09T02:19:43.000Z","updated_at":"2024-02-09T04:43:55.000Z","dependencies_parsed_at":"2022-08-30T07:51:33.035Z","dependency_job_id":null,"html_url":"https://github.com/jprante/elasticsearch-knapsack","commit_stats":null,"previous_names":[],"tags_count":64,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jprante%2Felasticsearch-knapsack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jprante%2Felasticsearch-knapsack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jprante%2Felasticsearch-knapsack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jprante%2Felasticsearch-knapsack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jprante","download_url":"https://codeload.github.com/jprante/elasticsearch-knapsack/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245944019,"owners_count":20697945,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T18:01:03.310Z","updated_at":"2025-03-27T23:31:35.226Z","avatar_url":"https://github.com/jprante.png","language":"Java","funding_links":[],"categories":["Java","Elasticsearch developer tools and utilities","Elasticsearch"],"sub_categories":["Import and Export","Tools"],"readme":"![Knapsack](https://github.com/jprante/elasticsearch-knapsack/raw/master/src/site/resources/knapsack.jpg)\n\nImage by [Rick McCharles](https://www.flickr.com/photos/rickmccharles/3820820609)\nCC BY 2.0 https://creativecommons.org/licenses/by/2.0/\n\n# Knapsack plugin for Elasticsearch\n\nKnapsack is an \"swiss knife\" export/import plugin for [Elasticsearch](http://github.com/elasticsearch/elasticsearch).\nIt uses archive formats (tar, zip, cpio) and also Elasticsearch bulk format with \ncompression algorithms (gzip, bzip2, lzf, xz).\n\nA pull or push of indexes or search hits with stored fields across clusters is also supported.\n\nThe knapsack actions can be executed via HTTP REST, or in Java using the Java API.\n\nIn archive files, the following index information is encoded:\n\n- index settings\n- index mappings\n- index aliases\n\nWhen importing archive files again, this information is reapplied.\n\n## Compatibility matrix\n\n![Travis](https://travis-ci.org/jprante/elasticsearch-knapsack.png)\n\n| Elasticsearch  |   Plugin       | Release date |\n| -------------- | -------------- | ------------ |\n| 2.3.4          | 2.3.4.0        | Aug  4, 2016 |\n| 2.3.3          | 2.3.3.0        | May 23, 2016 |\n| 2.3.1          | 2.3.1.0        | Apr 21, 2016 |\n| 2.3.0          | 2.3.0.0        | Mar 31, 2016 |\n| 2.2.1          | 2.2.1.0        | Mar 31, 2016 |\n| 2.1.2          | 2.1.2.0        | Mar 23, 2016 |\n| 2.2.0          | 2.2.0.0        | Feb 23, 2016 |\n| 2.1.1          | 2.1.1.0        | Dec 30, 2015 |\n| 2.1.0          | 2.1.0.0        | Dec  7, 2015 |\n| 2.0.0          | 2.0.0.0        | Nov 14, 2015 |\n| 2.0.0-rc1      | 2.0.0-rc1.0    | Oct 12, 2015 |\n\nFor older releases and 1.x versions, see the repective branches.\n\n## Installation 2.x\n\n    ./bin/plugin install http://xbib.org/repository/org/xbib/elasticsearch/plugin/elasticsearch-knapsack/2.3.4.0/elasticsearch-knapsack-2.3.4.0-plugin.zip\n\nDo not forget to restart the node after installation.\n\nNote: If you get an error while exporting or importing like this\n\n    {\"error\":{\"root_cause\":[{\"type\":\"access_control_exception\",\"reason\":\"access denied (\\\"java.io.FilePermission\\\" \\\"/foo/bar.zip\\\" \\\"read\\\")\"}],\"type\":\"access_control_exception\",\"reason\":\"access denied (\\\"java.io.FilePermission\\\" \\\"/foo/bar.zip\\\" \\\"read\\\")\"},\"status\":500}\n\nthen you are blocked by the Elasticsearch 2.x security manager. In this case, choose another directory for reading/writing archive files, preferably `path.logs`.\n\nIt is recommended to add a node with knapsack plugin installed only, no data, no master, and removing the node after the export/import completed.\n \n## Project docs\n\nThe Maven project site is available at [Github](http://jprante.github.io/elasticsearch-knapsack)\n\n## Overview\n\n![Diagram](https://github.com/jprante/elasticsearch-knapsack/raw/master/src/site/resources/knapsack-diagram-2.png)\n\n# Example\n\nLet's go through a simple example:\n\n    curl -XDELETE localhost:9200/test\n    curl -XPUT localhost:9200/test/test/1 -d '{\"key\":\"value 1\"}'\n    curl -XPUT localhost:9200/test/test/2 -d '{\"key\":\"value 2\"}'\n\n# Export\n\nYou can export this Elasticsearch index with\n\n    curl -XPOST localhost:9200/test/test/_export\n    {\"running\":true,\"state\":{\"mode\":\"export\",\"started\":\"2015-10-12T18:13:47.214Z\",\"path\":\"file:///Users/es/elasticsearch-2.0.0-rc1/logs/_all.tar.gz\",\"node_name\":\"Doctor Bong\"}}\n\nThe result is a file in the Elasticsearch `path.logs` folder\n\n    -rw-r--r--   1 joerg  staff          343 28 Sep 21:18 test_test.tar.gz\n   \nCheck with tar utility, the settings and the mapping is also exported\n\n    tar ztvf test_test.tar.gz \n    -rw-r--r--  0 joerg  0         133 28 Sep 21:18 test/_settings/null/null\n    -rw-r--r--  0 joerg  0          49 28 Sep 21:18 test/test/_mapping/null\n    -rw-r--r--  0 joerg  0          17 28 Sep 21:18 test/test/1/_source\n    -rw-r--r--  0 joerg  0          17 28 Sep 21:18 test/test/2/_source\n\nAlso, you can export a whole index with\n\n    curl -XPOST localhost:9200/test/_export\n\nwith the archive file `test.tar.gz`, or even all cluster indices with\n\n    curl -XPOST 'localhost:9200/_export'\n\nto the file `_all.tar.gz`\n\n## Available suffixes for archive formats\n\n    .tar\n    .zip\n    .cpio\n    .bulk\n\n## Available suffixes for compression\n\n    .gz\n    .bzip2\n    .xz\n    .lzf\n\nBy default, the archive format is `tar` with compression `gz` (gzip). \n\nYou can also export to `zip`, `cpio` or `bulk` archive format.\n\nAvailable compression codecs are `bz2` (bzip2), `xz` (Xz), or `lzf` (LZF)\n\nNote: if you use the `bulk` format, you create Elasticsearch bulk format.\n\n## Export search results\n\nYou can add a query to the `_export` endpoint just like you would do for searching in Elasticsearch.\n\n    curl -XPOST 'localhost:9200/test/test/_export' -d '{\n       \"query\" : {\n           \"match_phrase\" : {\n               \"key\" : \"value 1\"\n           }\n       },\n       \"fields\" : [ \"_parent\", \"_source\" ]\n    }'\n\n## Export to an archive with a given archive path name\n\nYou can configure an archive path with the parameter `archivepath`\n\n    curl -XPOST 'localhost:9200/test/_export?archivepath=/tmp/myarchive.zip'\n\nIf Elasticsearch can not write to the archive path, an error message will appear, and no export will take place.\n\nNote: Elasticsearch 2.x has a security manager enabled by default which prevents reading/writing to locations outside of \nElasticsearch directories.\nTherefore, the default location for export/import is set to the `path.logs` directory.\nIf you prefer to write to or read from any locations, you can disable the security manager by\n\n    ./bin/elasticsearch ... -Dsecurity.manager.enabled=false\n\nExisting archive files are not overwritten. You can force overwrite with the parameter `overwrite=true`\n\n## Export split by byte size\n\nYou can create multiple archive files with the parameter `bytes`\n\n    curl -XPOST 'localhost:9200/test/_export?archivepath=/tmp/myindex.bulk\u0026bytes=10m'\n\nThis creates `myindex.bulk`, `1.myindex.bulk`, `2.myindex.bulk` ... where all archive files are around 10 megabytes.\n\n## Renaming indexes and index types\n\nYou can rename indexes and index types by adding a `map` parameter that contains a JSON\nobject with old and new index (and index/type) names.\n\n    curl -XPOST 'localhost:9200/test/type/_export?map=\\{\"test\":\"testcopy\",\"test/type\":\"testcopy/typecopy\"\\}'\n\nNote the backslash, which is required to escape shell interpretation of curly braces.\n\n## Push or pull indices from one cluster to another\n\nIf you want tp push or pull indices from one cluster to another, Knapsack is your friend.\n\nYou can copy an index in the local cluster or to a remote cluster with the `_push` or the `_pull` endpoint.\nThis works if you have the same Java JVM version and the same Elasticsearch version.\n\nExample for a local cluster copy of the index `test` to `testcopy`\n\n    curl -XPOST 'localhost:9200/test/_push?map=\\{\"test\":\"testcopy\"\\}'\n\nExample for a remote cluster copy of the index `test` by using the parameters `cluster`, `host`, and `port`\n\n    curl -XPOST 'localhost:9200/test/_push?\u0026cluster=remote\u0026host=127.0.0.1\u0026port=9201'\n\nThis is a complete example that illustrates how to filter an index by timestamp and copy this part to\nanother index\n\n    curl -XDELETE 'localhost:9200/test'\n    curl -XDELETE 'localhost:9200/testcopy'\n    curl -XPUT 'localhost:9200/test/' -d '\n    {\n        \"mappings\" : {\n            \"_default_\": {\n                \"_timestamp\" : { \"enabled\" : true, \"store\" : true, \"path\" : \"date\" }\n            }\n        }\n    }\n    '\n    curl -XPUT 'localhost:9200/test/doc/1' -d '\n    {\n        \"date\" : \"2014-01-01T00:00:00\",\n        \"sentence\" : \"Hi!\",\n        \"value\" : 1\n    }\n    '\n    curl -XPUT 'localhost:9200/test/doc/2' -d '\n    {\n        \"date\" : \"2014-01-02T00:00:00\",\n        \"sentence\" : \"Hello World!\",\n        \"value\" : 2\n    }\n    '\n    curl -XPUT 'localhost:9200/test/doc/3' -d '\n    {\n        \"date\" : \"2014-01-03T00:00:00\",\n        \"sentence\" : \"Welcome!\",\n        \"value\" : 3\n    }\n    '\n    curl 'localhost:9200/test/_refresh'\n    curl -XPOST 'localhost:9200/test/_push?map=\\{\"test\":\"testcopy\"\\}' -d '\n    {\n        \"fields\" : [ \"_timestamp\", \"_source\" ],\n        \"query\" : {\n             \"filtered\" : {\n                 \"query\" : {\n                     \"match_all\" : {\n                     }\n                 },\n                 \"filter\" : {\n                    \"range\": {\n                       \"_timestamp\" : {\n                           \"from\" : \"2014-01-02\"\n                       }\n                    }\n                 }\n             }\n         }\n    }\n    '\n    curl '0:9200/test/_search?fields=_timestamp\u0026pretty'\n    curl '0:9200/testcopy/_search?fields=_timestamp\u0026pretty'\n\n# Import\n\nYou can import the file with the `_import` endpoint\n\n    curl -XPOST 'localhost:9200/test/test/_import'\n\nKnapsack does not delete or overwrite data by default.\nBut you can use the parameter `createIndex` with the value `false` to allow indexing to indexes that exist.\n\nWhen importing, you can map your indexes or index/types to your favorite ones.\n\n    curl -XPOST 'localhost:9200/test/_import?map=\\{\"test\":\"testcopy\"\\}'\n\n## Modifying settings and mappings\n\nYou can overwrite the settings and mapping when importing by using parameters in the form \n`\u003cindex\u003e_settings=\u003cfilename\u003e` or `\u003cindex\u003e_\u003ctype\u003e_mapping=\u003cfilename\u003e`. \n\nGeneral example::\n\n    curl -XPOST 'localhost:9200/myindex/mytype/_import?myindex_settings=/my/new/mysettings.json\u0026myindex_mytype_mapping=/my/new/mapping.json'\n\nThe following statements demonstrate how you can change the number of shards from the default `5` to `1` \nand replica from `1` to `0` for an index `test`\n\n    curl -XDELETE localhost:9200/test\n    curl -XPUT 'localhost:9200/test/test/1' -d '{\"key\":\"value 1\"}'\n    curl -XPUT 'localhost:9200/test/test/2' -d '{\"key\":\"value 2\"}'\n    curl -XPUT 'localhost:9200/test2/foo/1' -d '{\"key\":\"value 1\"}'\n    curl -XPUT 'localhost:9200/test2/bar/1' -d '{\"key\":\"value 1\"}'\n    curl -XPOST 'localhost:9200/test/_export'\n    tar zxvf test.tar.gz test/_settings\n    echo '{\"index.number_of_shards\":\"1\",\"index.number_of_replicas\":\"0\"}' \u003e test/_settings/null/null\n    curl -XDELETE 'localhost:9200/test'\n    curl -XPOST 'localhost:9200/test/_import?test_settings=test/_settings/null/null'\n    curl -XGET 'localhost:9200/test/_settings?pretty'\n    curl -XPOST 'localhost:9200/test/_search?q=*\u0026pretty'\n\nThe result is a search on an index with just one shard.\n\n    {\n      \"took\" : 19,\n      \"timed_out\" : false,\n      \"_shards\" : {\n        \"total\" : 1,\n        \"successful\" : 1,\n        \"failed\" : 0\n      },\n      \"hits\" : {\n        \"total\" : 2,\n        \"max_score\" : 1.0,\n        \"hits\" : [ {\n          \"_index\" : \"test\",\n          \"_type\" : \"test\",\n          \"_id\" : \"1\",\n          \"_score\" : 1.0,\n          \"_source\":{\"key\":\"value 1\"}\n        }, {\n          \"_index\" : \"test\",\n          \"_type\" : \"test\",\n          \"_id\" : \"2\",\n          \"_score\" : 1.0,\n          \"_source\":{\"key\":\"value 2\"}\n        } ]\n      }\n    }\n\n## State of knapsack import/export actions\n\nWhile exports or imports or running, you can check the state with\n\n    curl -XPOST 'localhost:9200/_export/state'\n\nor\n\n    curl -XPOST 'localhost:9200/_import/state'\n\n## Aborting knapsack actions\n\nIf you want to abort all running knapsack exports/import, you can do this by\n\n    curl -XPOST 'localhost:9200/_export/abort'\n\nor\n\n    curl -XPOST 'localhost:9200/_import/abort'\n\n# Handing Parent/Child documents\n\n## Exporting \n\nHandling dependant documents is bit tricky since indexing a child document requires the presence of its parent document. A simple approach is to export the documents into seperate archives by using a query. In case your child documents are located in the same type as the parent documents, define the appropriate filter in the query. If you have stored the child documents in a seperate type, you can export the type containing the parent documents like this:\n\n    curl -XPOST 'localhost:9200/myIndex/myParentDocs/_export?archivepath=/tmp/myIndex_myParentDocs.zip'\n\nWhen exporting the type containing the child documents, include the \"_parent\" meta field\n\n    curl -XPOST 'localhost:9200/myIndex/myChildDocs/_export?archivepath=/tmp/myIndex_myChildDocs.zip'' -d '{\n       \"query\" : {\n           \"match_all\" : {\n           }\n       },\n       \"fields\" : [ \"_parent\", \"_source\" ]\n    }'\n\n\n## Importing Parent/Child documents\n\nBefore you import the parent documents, you have to create the index manually first: Each type export only contains the mapping of that spedific type and you cannot add a dependant mapping in a second step later. All dependant mappings must be created at the same time otherwise you'll get an error like \"java.lang.IllegalArgumentException: Can't specify parent if no parent field has been configured\". After creating the index, import the parent documents:\n\n    curl -XPOST 'localhost:9200/myIndex/myParentDocs/_import?archivepath=/tmp/myIndex_myParentDocs.zip\u0026createIndex=false'\n\nThen import the child documents:\n\n    curl -XPOST 'localhost:9200/myIndex/myChildDocs/_import?archivepath=/tmp/myIndex_myChildDocs.zip\u0026createIndex=false'\n    \nRepeat this for all your child types.\n\n# Java API\n\nKnapsack implements all actions as Java transport actions in Elasticsearch.\n\nYou can consult the junit tests for finding out how to use the API. To give you an impression, \nhere is just an example for a very minimal export/import cycle using the `bulk` archive format.\n\n        \n        client.index(new IndexRequest().index(\"index1\").type(\"test1\").id(\"doc1\")\n             .source(\"content\",\"Hello World\").refresh(true)).actionGet();\n        \n        File exportFile = File.createTempFile(\"minimal-import-\", \".bulk\");\n        Path exportPath = Paths.get(URI.create(\"file:\" + exportFile.getAbsolutePath()));\n        KnapsackExportRequestBuilder requestBuilder = new KnapsackExportRequestBuilder(client.admin().indices())\n                .setArchivePath(exportPath)\n                .setOverwriteAllowed(false);\n        KnapsackExportResponse knapsackExportResponse = requestBuilder.execute().actionGet();\n\n        KnapsackStateRequestBuilder knapsackStateRequestBuilder =\n               new KnapsackStateRequestBuilder(client.admin().indices());\n        KnapsackStateResponse knapsackStateResponse = knapsackStateRequestBuilder.execute().actionGet();\n\n        Thread.sleep(1000L);\n\n        client.admin().indices().delete(new DeleteIndexRequest(\"index1\")).actionGet();\n\n        KnapsackImportRequestBuilder knapsackImportRequestBuilder = new KnapsackImportRequestBuilder(client.admin().indices())\n                .setArchivePath(exportPath);\n        KnapsackImportResponse knapsackImportResponse = knapsackImportRequestBuilder.execute().actionGet();\n\n# Caution\n\nKnapsack is very simple and works without locks or snapshots. This means, if Elasticsearch is\nallowed to write to the part of your data in the export while it runs, you may lose data in the export.\nSo it is up to you to organize the safe export and import with this plugin.\n\nIf you want a more advanced feature, please use the snapshot/restore which is the standard\nprocedure for saving/restoring data in Elasticsearch:\n\nhttp://www.elasticsearch.org/blog/introducing-snapshot-restore/\n\n# Credits\n\nKnapsack contains derived work of Apache Common Compress\nhttp://commons.apache.org/proper/commons-compress/\n\nThe code in this component has many origins:\nThe bzip2, tar and zip support came from Avalon's Excalibur, but originally\nfrom Ant, as far as life in Apache goes. The tar package is originally Tim Endres'\npublic domain package. The bzip2 package is based on the work done by Keiron Liddle as\n well as Julian Seward's libbzip2. It has migrated via:\nAnt -\u003e Avalon-Excalibur -\u003e Commons-IO -\u003e Commons-Compress.\nThe cpio package is based on contributions of Michael Kuss and the jRPM project.\n\nThanks to `nicktgr15 \u003chttps://github.com/nicktgr15\u003e` for extending Knapsack to support Amazon S3.\n\n# License\n\nKnapsack Plugin for Elasticsearch\n\nCopyright (C) 2012 Jörg Prante\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjprante%2Felasticsearch-knapsack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjprante%2Felasticsearch-knapsack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjprante%2Felasticsearch-knapsack/lists"}