{"id":15044426,"url":"https://github.com/elastic/elasticsearch-river-couchdb","last_synced_at":"2025-04-07T18:14:22.565Z","repository":{"id":1984871,"uuid":"2916813","full_name":"elastic/elasticsearch-river-couchdb","owner":"elastic","description":"CouchDB River Plugin for elasticsearch (STOPPED)","archived":false,"fork":false,"pushed_at":"2024-08-08T07:13:41.000Z","size":206,"stargazers_count":162,"open_issues_count":0,"forks_count":57,"subscribers_count":354,"default_branch":"master","last_synced_at":"2025-03-30T22:13:48.725Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/elastic.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2011-12-05T13:57:37.000Z","updated_at":"2025-03-18T21:49:02.000Z","dependencies_parsed_at":"2024-09-25T02:03:38.800Z","dependency_job_id":null,"html_url":"https://github.com/elastic/elasticsearch-river-couchdb","commit_stats":{"total_commits":116,"total_committers":15,"mean_commits":7.733333333333333,"dds":0.3448275862068966,"last_synced_commit":"dc669a7dd5316fdcc1c33d4fa58d7290451b88b1"},"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Felasticsearch-river-couchdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Felasticsearch-river-couchdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Felasticsearch-river-couchdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Felasticsearch-river-couchdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/elastic","download_url":"https://codeload.github.com/elastic/elasticsearch-river-couchdb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247704571,"owners_count":20982298,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-24T20:50:34.386Z","updated_at":"2025-04-07T18:14:22.549Z","avatar_url":"https://github.com/elastic.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"**Important**: This project has been stopped since elasticsearch 2.0.\n\n----\n\nCouchDB River Plugin for Elasticsearch\n==================================\n\nThe CouchDB River plugin allows to automatically index couchdb and make it searchable using the excellent \n[_changes](http://guide.couchdb.org/draft/notifications.html) stream couchdb provides.\n\n**Rivers are [deprecated](https://www.elastic.co/blog/deprecating_rivers) and will be removed in the future.**\nHave a look at [logstash couchdb changes input](http://www.elastic.co/guide/en/logstash/current/plugins-inputs-couchdb_changes.html).\n\nIn order to install the plugin, run: \n\n```sh\nbin/plugin install elasticsearch/elasticsearch-river-couchdb/2.6.0\n```\n\nYou need to install a version matching your Elasticsearch version:\n\n|       Elasticsearch    |CouchDB River Plugin|                                                             Docs                                                                   |\n|------------------------|--------------------|------------------------------------------------------------------------------------------------------------------------------------|\n|    master              | Build from source  | See below                                                                                                                          |\n|    es-1.x              | Build from source  | [2.7.0-SNAPSHOT](https://github.com/elasticsearch/elasticsearch-river-couchdb/tree/es-1.x/#version-270-snapshot-for-elasticsearch-1x)|\n|    es-1.6              |     2.6.0         | [2.6.0](https://github.com/elastic/elasticsearch-river-couchdb/tree/v2.6.0/#version-260-for-elasticsearch-16)                  |\n|    es-1.5              |     2.5.0         | [2.5.0](https://github.com/elastic/elasticsearch-river-couchdb/tree/v2.5.0/#version-250-for-elasticsearch-15)                  |\n|    es-1.4              |     2.4.2         | [2.4.2](https://github.com/elasticsearch/elasticsearch-river-couchdb/tree/v2.4.2/#version-242-for-elasticsearch-14)                  |\n|    es-1.3              |     2.3.0         | [2.3.0](https://github.com/elasticsearch/elasticsearch-river-couchdb/tree/v2.3.0/#version-230-for-elasticsearch-13)                  |\n|    es-1.2              |     2.2.0          | [2.2.0](https://github.com/elasticsearch/elasticsearch-river-couchdb/tree/v2.2.0/#couchdb-river-plugin-for-elasticsearch)          |\n|    es-1.0              |     2.0.0          | [2.0.0](https://github.com/elasticsearch/elasticsearch-river-couchdb/tree/v2.0.0/#couchdb-river-plugin-for-elasticsearch)          |\n|    es-0.90             |     1.3.0          | [1.3.0](https://github.com/elasticsearch/elasticsearch-river-couchdb/tree/v1.3.0/#couchdb-river-plugin-for-elasticsearch)          |\n\nTo build a `SNAPSHOT` version, you need to build it with Maven:\n\n```bash\nmvn clean install\nplugin --install river-couchdb \\ \n       --url file:target/releases/elasticsearch-river-couchdb-X.X.X-SNAPSHOT.zip\n```\n\nCreate river\n------------\n\n Setting it up is as simple as executing the following against elasticsearch:\n\n```sh\ncurl -XPUT 'localhost:9200/_river/my_db/_meta' -d '{\n    \"type\" : \"couchdb\",\n    \"couchdb\" : {\n        \"host\" : \"localhost\",\n        \"port\" : 5984,\n        \"db\" : \"my_db\",\n        \"filter\" : null\n    },\n    \"index\" : {\n        \"index\" : \"my_db\",\n        \"type\" : \"my_db\",\n        \"bulk_size\" : \"100\",\n        \"bulk_timeout\" : \"10ms\"\n    }\n}'\n```\n\nThis call will create a river that uses the `_changes` stream to index all data within couchdb. Moreover, any \"future\" changes will automatically be indexed as well, making your search index and couchdb synchronized at all times.\n\nThe couchdb river is provided as a [plugin](https://github.com/elasticsearch/elasticsearch-river-couchdb) (including explanation on how to install it).\n\nOn top of that, in case of a failover, the couchdb river will automatically be started on another elasticsearch node, and continue indexing from the last indexed seq.\n\nBulking\n======\n\nBulking is automatically done in order to speed up the indexing process. If within the specified `bulk_timeout` more changes are detected,\nchanges will be bulked up to `bulk_size` before they are indexed.\n\nSince 1.3.0, by default, `bulk` size is `100`. A bulk is flushed every `5s`. Number of concurrent requests allowed to be executed is 1.\nYou can modify those settings within index section:\n\n```javascript\n{\n    \"type\" : \"couchdb\",\n    \"index\" : {\n        \"index\" : \"my_index\",\n        \"type\" : \"my_type\",\n        \"bulk_size\" : 1000,\n        \"flush_interval\" : \"1s\",\n        \"max_concurrent_bulk\" : 3\n    }\n}\n```\n\nFiltering\n======\n\nThe `changes` stream allows to provide a filter with parameters that will be used by couchdb to filter the stream of changes. Here is how it can be configured:\n\n```javascript\n{\n    \"couchdb\" : {\n        \"filter\" : \"test\",\n        \"filter_params\" : {\n            \"param1\" : \"value1\",\n            \"param2\" : \"value2\"\n        }\n    }\n}\n```\n\nScript Filters\n=========\n\nFiltering can also be performed by providing a script that will further process each changed item \nwithin the changes stream. The json provided to the script is under a var called **ctx** with the relevant seq stream \nchange (for example, **ctx.doc** will refer to the document, or **ctx.deleted** is the flag if its deleted or not).\n\nAny other [script language supported by Elasticsearch](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-plugins.html#scripting) \nmay be used by setting the `script_type` parameter to the appropriate value. \n\nIf unspecified, the default is `groovy`. \nSee [Scripting documentation](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html) for details. \n\nThe **ctx.doc** can be changed and its value can will be indexed (assuming its not a deleted change). \nAlso, if **ctx.ignore** is set to true, the change seq will be ignore and not applied.\n\nOther possible values that can be set are **ctx.index** to control the index name to index the doc into, **ctx.type** \nto control the (mapping) type to index into, **ctx._parent** and **ctx._routing**.\n\nHere is an example setting that adds `field1` with value `value1` to all docs:\n\n```javascript\n{\n    \"type\" : \"couchdb\",\n    \"couchdb\" : {\n        \"script\" : \"ctx.doc.field1 = 'value1'\"\n    }\n}\n```\n\nBasic Authentication\n===============\n\nBasic Authentication can be used by passing the **user** and **password** attributes.\n\n```javascript\n{\n    \"type\" : \"couchdb\",\n    \"couchdb\" : {\n        \"user\" : \"alice\",\n        \"password\" : \"secret\"\n    }\n}\n```\n\nHTTPS\n=====\n\nTo use HTTPS, pass the **protocol** field. Most likely, you will also have to change the **port**. If you have unfixable problems with the servers certificates for any reason, you can disable hostname verification by passing **no_verify**.\n\n```javascript\n{\n    \"type\" : \"couchdb\",\n    \"couchdb\" : {\n        \"protocol\" : \"https\",\n        \"port\" : 443,\n        \"no_verify\" : \"true\"\n    }\n}\n```\n\n\nIgnoring Attachments\n====================\n\nYou can ignore attachments as provided by couchDb for each document (`_attachments` field).\n\nHere is an example setting that disable *attachments* for all docs:\n\n```javascript\n{\n  \"type\":\"couchdb\",\n  \"couchdb\": {\n    \"ignore_attachments\":true\n  }\n}\n```\n\n\nNote, by default, attachments are not ignored (**false**)\n\n\nHeartbeat\n=========\n\nBy default, couchdb river set _changes API heartbeat to `10s`.\nSince 1.3.0, an additional option has been added to control the HTTP connection timeout (default to `30s`).\nyou can control both settings using `heartbeat` and `read_timeout` options:\n\n```sh\ncurl -XPUT 'localhost:9200/_river/my_db/_meta' -d '{\n    \"type\" : \"couchdb\",\n    \"couchdb\" : {\n        \"host\" : \"localhost\",\n        \"port\" : 5984,\n        \"db\" : \"my_db\",\n        \"heartbeat\" : \"5s\",\n        \"read_timeout\" : \"15s\"\n    }\n}'\n```\n\nStarting at a Specific Sequence\n==========\n\nThe CouchDB river stores the `last_seq` value in a document called `_seq` in the `_river` index. You can use this fact to start or resume rivers at a particular sequence.\n\nTo have the CouchDB river start at a particular `last_seq`, create a document with contents like this:\n\n````sh\ncurl -XPUT 'localhost:9200/_river/my_db/_seq' -d '\n{\n  \"couchdb\": {\n    \"last_seq\": \"100\"\n  }\n}'\n````\n\nwhere 100 is the sequence number you want the river to start from. Then create the `_meta` document as before. The CouchDB river will startup and read the last sequence value and start indexing from there.\n\n\nExamples\n========\n\n\nIndexing Databases with Multiple Types\n-------------------------------------\n\nA common pattern in CouchDB is to have a single database hold documents\nof multiple types. Typically each document will have a field containing\nthe type of the document.\n\nFor example, a database of products from Amazon might have a book type:\n\n```json\n{\n  \"_id\": 1,\n  \"type\" : \"book\",\n  \"author\" : \"Michael McCandless\",\n  \"title\" : \"Lucene in Action\"\n}\n```\n\nand a CD type:\n\n```json\n{\n  \"_id\": 2,\n  \"type\" : \"cd\",\n  \"artist\" : \"Tool\",\n  \"title\" : \"Undertow\"\n}\n```\n\nElasticsearch also supports multiple types in the same index and we need\nto tell ES where each type from CouchDB goes. You do this with a river\ndefinition like this:\n\n```json\n{\n  \"type\" : \"couchdb\",\n  \"couchdb\" : {\n    \"host\" : \"localhost\",\n    \"port\" : 5984,\n    \"db\" : \"amazon\",\n    \"script\" : \"ctx._type = ctx.doc.type\"\n  },\n  \"index\" : {\n    \"index\" : \"amazon\"\n  }\n}\n```\n\nSetting  `ctx._type` tells Elasticsearch what type in the index to use.\nSo if doc.type for a CouchDB changeset is \"book\" then Elasticsearch will\nindex it as the \"book\" type.\n\nThe script block can be expanded to handle more complicated cases if\nyour types don't map one-to-one between the systems.\n\nIf you need to also handle deleting documents from the right type in\nElasticsearch, be aware that the above setup requires you to delete\ndocuments from CouchDB in a special way. If you use the `DELETE` `HTTP`\nverb with CouchDB you will break the above river as ES will be unable\nto determine what type you're trying to delete. Instead you must\npreserve some information about the document you deleted by using the\nCouchDB bulk document interface.\n\nFor example, to delete the above \"cd\" document, you must post a document\nlike this to the CouchDB server, replacing the `_rev` field with the\nrevision of the document you want to delete:\n\n```sh\ncurl -XPOST http://localhost:5984/amazon/_bulk_docs -d '\n{\n  \"docs\" : [\n    {\n        \"_id: 2,\n        \"_rev\" : \"rev\",\n        \"_deleted\" : true,\n        \"type\" : \"cd\"\n    }\n  ]\n}'\n```\n\nThis deletes the document while preserving the type information for ES.\nYou can extend this technique to store more data in deleted documents\nbut be aware of the disk space usage.\n\nIndexing parent/child documents\n-------------------------------\n\nIf you need to index relational documents using the parent/child feature, you could\ndo it using a script filter.\n\nFor example, let's say you have two types of document in CouchDB: Regions and Campuses.\n\n```\n// Region 1\n{\n    \"type\": \"region\",\n    \"name\": \"bretagne\"\n}\n```\n\n```\n// Campus 2\n{\n    \"type\": \"campus\",\n    \"name\": \"enib\",\n    \"parent_id\": \"1\"\n}\n```\n\nYou can use the following mapping for `campus`:\n\n```\n{\n    \"campus\" : {\n        \"_parent\" : {\n            \"type\" : \"region\"\n        }\n    }\n}\n```\n\nAnd the launch the river with the following script:\n\n```\n{\n    \"type\": \"couchdb\",\n    \"couchdb\": {\n        \"script\": \"ctx._type = ctx.doc.type; if (ctx._type == 'campus') { ctx._parent = ctx.doc.parent_id; }\"\n    }\n}\n```\n\nRemoving fields using a script\n------------------------------\n\nYou can remove fields using a script like this:\n\n```\n{\n    \"type\": \"couchdb\",\n    \"couchdb\": {\n        \"script\": \"var oblitertron = function(x) { var things = [\\\"foo\\\"]; var toberemoved = new java.util.ArrayList(); foreach (i : x.keySet()) { if(things.indexOf(i) == -1) { toberemoved.add(i); } } foreach (i : toberemoved) { x.remove(i); } return x; }; ctx.doc = oblitertron(ctx.doc);\"\n    }\n}\n```\n\nA more readable version of the script (with comments) is:\n\n```js\nvar oblitertron = function (x) {\n    // List of fields we want to keep. Others will be removed, such as _id, _rev...\n    var things = [\"foo\"];\n    var toberemoved = new java.util.ArrayList();\n    foreach(i : x.keySet()) {\n        if (things.indexOf(i) == -1) {\n            // If we find a field to be removed, we store its name in an array\n            toberemoved.add(i);\n        }\n    }\n    // We remove useless fields\n    foreach(i : toberemoved) {\n        x.remove(i);\n    }\n    // We return the new document which will be indexed.\n    return x;\n};\nctx.doc = oblitertron(ctx.doc);\n```\n\n\nTests\n=====\n\nTo run couchdb integration tests, you need to have couchdb running on `localhost` using default `5984` port.\nThen you can run tests using `tests.couchdb` option:\n\n```sh\nmvn clean test -Dtests.couchdb=true\n```\n\n\nLicense\n=======\n\n    This software is licensed under the Apache 2 license, quoted below.\n\n    Copyright 2009-2014 Elasticsearch \u003chttp://www.elasticsearch.org\u003e\n\n    Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n    use this file except in compliance with the License. You may obtain a copy of\n    the License at\n\n        http://www.apache.org/licenses/LICENSE-2.0\n\n    Unless required by applicable law or agreed to in writing, software\n    distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT\n    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the\n    License for the specific language governing permissions and limitations under\n    the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felastic%2Felasticsearch-river-couchdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Felastic%2Felasticsearch-river-couchdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felastic%2Felasticsearch-river-couchdb/lists"}