{"id":15151933,"url":"https://github.com/elastic/elasticsearch-river-rabbitmq","last_synced_at":"2025-09-29T22:31:16.035Z","repository":{"id":1984936,"uuid":"2916880","full_name":"elastic/elasticsearch-river-rabbitmq","owner":"elastic","description":"RabbitMQ River Plugin for elasticsearch (STOPPED)","archived":true,"fork":false,"pushed_at":"2023-03-14T17:35:08.000Z","size":988,"stargazers_count":174,"open_issues_count":0,"forks_count":83,"subscribers_count":374,"default_branch":"master","last_synced_at":"2025-01-19T16:54:24.255Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/elastic.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2011-12-05T14:10:21.000Z","updated_at":"2024-03-31T14:12:26.000Z","dependencies_parsed_at":"2023-07-05T16:00:18.244Z","dependency_job_id":null,"html_url":"https://github.com/elastic/elasticsearch-river-rabbitmq","commit_stats":{"total_commits":132,"total_committers":17,"mean_commits":7.764705882352941,"dds":"0.43181818181818177","last_synced_commit":"05e886103918949d3c329ce803bdfb9de74dea51"},"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Felasticsearch-river-rabbitmq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Felasticsearch-river-rabbitmq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Felasticsearch-river-rabbitmq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Felasticsearch-river-rabbitmq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/elastic","download_url":"https://codeload.github.com/elastic/elasticsearch-river-rabbitmq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234673078,"owners_count":18869582,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-26T15:40:40.719Z","updated_at":"2025-09-29T22:31:15.713Z","avatar_url":"https://github.com/elastic.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"**Important**: This project has been stopped since elasticsearch 2.0.\n\n----\n\nRabbitMQ River Plugin for Elasticsearch\n==================================\n\nThe RabbitMQ River plugin allows index [bulk format messages](http://www.elasticsearch.org/guide/reference/api/bulk/) into elasticsearch.\nRabbitMQ River allows to automatically index a [RabbitMQ](http://www.rabbitmq.com/) queue. The format of the messages follows the bulk api format:\n\n```javascript\n{ \"index\" : { \"_index\" : \"twitter\", \"_type\" : \"tweet\", \"_id\" : \"1\" } }\n{ \"tweet\" : { \"text\" : \"this is a tweet\" } }\n{ \"delete\" : { \"_index\" : \"twitter\", \"_type\" : \"tweet\", \"_id\" : \"2\" } }\n{ \"create\" : { \"_index\" : \"twitter\", \"_type\" : \"tweet\", \"_id\" : \"1\" } }\n{ \"tweet\" : { \"text\" : \"another tweet\" } }\n```\n\n**Rivers are [deprecated](https://www.elastic.co/blog/deprecating_rivers) and will be removed in the future.**\nHave a look at [logstash rabbitmq input](http://www.elastic.co/guide/en/logstash/current/plugins-inputs-rabbitmq.html).\n\nIn order to install the plugin, run: \n\n```sh\nbin/plugin install elasticsearch/elasticsearch-river-rabbitmq/2.6.0\n```\n\nYou need to install a version matching your Elasticsearch version:\n\n|       Elasticsearch    | RabbitMQ River    |                                                             Docs                                                                   |\n|------------------------|-------------------|------------------------------------------------------------------------------------------------------------------------------------|\n|    master              | Build from source | See below                                                                                                                          |\n|    es-1.x              | Build from source  | [2.7.0-SNAPSHOT](https://github.com/elasticsearch/elasticsearch-river-rabbitmq/tree/es-1.x/#version-270-snapshot-for-elasticsearch-1x)|\n|    es-1.6              |     2.6.0         | [2.6.0](https://github.com/elastic/elasticsearch-river-rabbitmq/tree/v2.6.0/#version-260-for-elasticsearch-16)                  |\n|    es-1.5              |     2.5.0         | [2.5.0](https://github.com/elastic/elasticsearch-river-rabbitmq/tree/v2.5.0/#version-250-for-elasticsearch-15)                  |\n|    es-1.4              |     2.4.1         | [2.4.1](https://github.com/elasticsearch/elasticsearch-river-rabbitmq/tree/v2.4.1/#version-241-for-elasticsearch-14)                  |\n|    es-1.3              |     2.3.0         | [2.3.0](https://github.com/elasticsearch/elasticsearch-river-rabbitmq/tree/v2.3.0/#version-230-for-elasticsearch-13)                  |\n|    es-1.2              |     2.2.0         | [2.2.0](https://github.com/elasticsearch/elasticsearch-river-rabbitmq/tree/v2.2.0/#version-220-for-elasticsearch-12)                  |\n|    es-1.1              |     2.0.0         | [2.0.0](https://github.com/elasticsearch/elasticsearch-river-rabbitmq/tree/v2.0.0/#rabbitmq-river-plugin-for-elasticsearch)  |\n|    es-1.0              |     2.0.0         | [2.0.0](https://github.com/elasticsearch/elasticsearch-river-rabbitmq/tree/v2.0.0/#rabbitmq-river-plugin-for-elasticsearch)  |\n|    es-0.90             |     1.6.0         | [1.6.0](https://github.com/elasticsearch/elasticsearch-river-rabbitmq/tree/v1.6.0/#rabbitmq-river-plugin-for-elasticsearch)  |\n\nTo build a `SNAPSHOT` version, you need to build it with Maven:\n\n```bash\nmvn clean install\nplugin --install river-rabbitmq \\ \n       --url file:target/releases/elasticsearch-river-rabbitmq-X.X.X-SNAPSHOT.zip\n```\n\nCreate river\n------------\n\nCreating the rabbitmq river is as simple as (all configuration parameters are provided, with default values):\n\n```sh\ncurl -XPUT 'localhost:9200/_river/my_river/_meta' -d '{\n    \"type\" : \"rabbitmq\",\n    \"rabbitmq\" : {\n        \"host\" : \"localhost\",\n        \"port\" : 5672,\n        \"user\" : \"guest\",\n        \"pass\" : \"guest\",\n        \"vhost\" : \"/\",\n        \"queue\" : \"elasticsearch\",\n        \"exchange\" : \"elasticsearch\",\n        \"routing_key\" : \"elasticsearch\",\n        \"exchange_declare\" : true,\n        \"exchange_type\" : \"direct\",\n        \"exchange_durable\" : true,\n        \"queue_declare\" : true,\n        \"queue_bind\" : true,\n        \"queue_durable\" : true,\n        \"queue_auto_delete\" : false,\n        \"heartbeat\" : \"30m\",\n        \"qos_prefetch_size\" : 0,\n        \"qos_prefetch_count\" : 10,\n        \"nack_errors\" : true\n    },\n    \"index\" : {\n        \"bulk_size\" : 100,\n        \"bulk_timeout\" : \"10ms\",\n        \"ordered\" : false,\n        \"replication\" : \"default\"\n    }\n}'\n```\n\nYou can disable exchange or queue declaration by setting `exchange_declare` or `queue_declare` to `false`\n(`true` by default).\nYou can disable queue binding by setting `queue_bind` to `false` (`true` by default).\n\nAddresses(host-port pairs) also available. it is useful to taking advantage rabbitmq HA(active/active) without any rabbitmq load balancer.\n(http://www.rabbitmq.com/ha.html)\n\n```javascript\n    ...\n    \"rabbitmq\" : {\n        \"addresses\" : [\n            {\n                \"host\" : \"rabbitmq-host1\",\n                \"port\" : 5672\n            },\n            {\n                \"host\" : \"rabbitmq-host2\",\n                \"port\" : 5672\n            }\n        ],\n        \"user\" : \"guest\",\n        \"pass\" : \"guest\",\n        \"vhost\" : \"/\",\n        ...\n    }\n    ...\n```\n\nThe river is automatically bulking queue messages if the queue is overloaded, allowing for faster catchup with the\nmessages streamed into the queue. The `ordered` flag allows to make sure that the messages will be indexed in the\nsame order as they arrive in the query by blocking on the bulk request before picking up the next data to be indexed.\nIt can also be used as a simple way to throttle indexing.\n\nYou can set `heartbeat` option to define heartbeat to RabbitMQ river even if no more messages are intended to be consumed\n(default to `30m`).\n\nReplication mode is set to node default value. You can change it by forcing `replication` to `async` or `sync`.\n\nBy default, when exception happens while executing bulk, failing messages are marked as rejected.\nYou can ignore errors and ack messages in any case setting `nack_errors` to `false`.\n\nSetting `qos_prefetch_size` will define maximum amount of content (measured in octets) that the server will deliver \n(0 if unlimited - default).\n\nSetting `qos_prefetch_count` will define maximum number of messages that the server will deliver (0 if unlimited). \nDefault to `bulk_size*2`.\n\nScripting\n---------\n\nRabbitMQ river can call scripts to modify or filter messages.\n\n### Full bulk scripting\n\nTo enable bulk scripting use the following configuration options:\n\n```sh\ncurl -XPUT 'localhost:9200/_river/my_river/_meta' -d '{\n    \"type\" : \"rabbitmq\",\n    \"rabbitmq\" : {\n        ...\n    },\n    \"index\" : {\n        ...\n    },\n    \"bulk_script_filter\" : {\n        \"script\" : \"myscript\",\n        \"script_lang\" : \"native\",\n        \"script_params\" : {\n            \"param1\" : \"val1\",\n            \"param2\" : \"val2\"\n            ...\n        }\n    }\n}'\n```\n\n* `script` is optional and is the name of the registered script in `elasticsearch.yml`. Basically, add the following\nproperty: `script.native.myscript.type: sample.MyNativeScriptFactory` and provide this class to elasticsearch\nclassloader.\n* `script_lang` is by default `native`.\n* `script_params` are optional configuration arguments for the script.\n\nThe script will receive a variable called `body` which contains a String representation of RabbitMQ's message body.\nThat `body` can be modified by the script, and it must return the new body as a String as well.\nIf the returned body is null, that message will be skipped from the indexing flow.\n\nFor more information see [Scripting module](http://www.elasticsearch.org/guide/reference/modules/scripting/)\n\n### Doc per doc scripting\n\nYou may also want to apply scripts document per document. It will only works for index or create operations.\n\nTo enable scripting use the following configuration options:\n\n```sh\ncurl -XPUT 'localhost:9200/_river/my_river/_meta' -d '{\n    \"type\" : \"rabbitmq\",\n    \"rabbitmq\" : {\n        ...\n    },\n    \"index\" : {\n        ...\n    },\n    \"script_filter\" : {\n        \"script\" : \"ctx.type1.field1 += param1\",\n        \"script_lang\" : \"mvel\",\n        \"script_params\" : {\n          \"param1\" : 1\n        }\n    }\n}'\n```\n\n* `script` is your javascript code if you use `mvel` scripts.\n* `script_lang` is by default `mvel`.\n* `script_params` are optional configuration arguments for the script.\n\nThe script will receive a variable called `ctx` which contains a String representation of the current document\nmeant to be indexed or created.\n\nFor more information see [Scripting module](http://www.elasticsearch.org/guide/reference/modules/scripting/)\n\nTests\n=====\n\nIntegrations tests in this plugin require working RabbitMQ service and therefore disabled by default. \nYou need to launch locally `rabbitmq-server` before starting integration tests.\n\nTo run test:\n\n```sh\nmvn clean test -Dtests.rabbitmq=true \n```\n\n\nLicense\n-------\n\n    This software is licensed under the Apache 2 license, quoted below.\n\n    Copyright 2009-2014 Elasticsearch \u003chttp://www.elasticsearch.org\u003e\n\n    Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n    use this file except in compliance with the License. You may obtain a copy of\n    the License at\n\n        http://www.apache.org/licenses/LICENSE-2.0\n\n    Unless required by applicable law or agreed to in writing, software\n    distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT\n    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the\n    License for the specific language governing permissions and limitations under\n    the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felastic%2Felasticsearch-river-rabbitmq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Felastic%2Felasticsearch-river-rabbitmq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felastic%2Felasticsearch-river-rabbitmq/lists"}