{"id":13500605,"url":"https://github.com/richardwilly98/elasticsearch-river-mongodb","last_synced_at":"2025-05-16T08:03:26.353Z","repository":{"id":2342108,"uuid":"3304447","full_name":"richardwilly98/elasticsearch-river-mongodb","owner":"richardwilly98","description":"MongoDB River Plugin for ElasticSearch","archived":false,"fork":false,"pushed_at":"2016-12-22T15:05:06.000Z","size":15131,"stargazers_count":1125,"open_issues_count":146,"forks_count":214,"subscribers_count":78,"default_branch":"master","last_synced_at":"2025-05-06T16:11:46.522Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/richardwilly98.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-01-30T11:42:26.000Z","updated_at":"2025-05-05T12:21:34.000Z","dependencies_parsed_at":"2022-09-16T14:11:14.247Z","dependency_job_id":null,"html_url":"https://github.com/richardwilly98/elasticsearch-river-mongodb","commit_stats":null,"previous_names":[],"tags_count":32,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/richardwilly98%2Felasticsearch-river-mongodb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/richardwilly98%2Felasticsearch-river-mongodb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/richardwilly98%2Felasticsearch-river-mongodb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/richardwilly98%2Felasticsearch-river-mongodb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/richardwilly98","download_url":"https://codeload.github.com/richardwilly98/elasticsearch-river-mongodb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254493381,"owners_count":22080126,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T22:01:06.821Z","updated_at":"2025-05-16T08:03:26.332Z","avatar_url":"https://github.com/richardwilly98.png","language":"Java","funding_links":[],"categories":["Java","II. Databases, search engines, big data and machine learning"],"sub_categories":["4. Client and drivers for databases"],"readme":"Issue Tracker\n==================================\n\nThis project is maintained in the spare time of contributors and so time able to be dedicated to it is extremely limited.\n\nPlease file issues only for reproducable problems. Given our limited time, we are not able to provide any help using the river and issues that simply say \"things aren't working\" will be closed. However, if you're able to diagnose an issue yourself we may be able to help with a fix and we are happy to review pull requests.\n\nTo debug issues, try changing the log level to trace as described in [the wiki](https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki). You may wish to build a custom version of the river with extra logging.\n\nMongoDB River Plugin for ElasticSearch\n==================================\n\nThis plugin uses MongoDB or the TokuMX fork of MongoDB as datasource to store data in ElasticSearch.\nFiltering and transformation are also possible. \nSee the [wiki](https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki) for more details.\n\nIn order to install the plugin, simply run: ```bin/plugin --install com.github.richardwilly98.elasticsearch/elasticsearch-river-mongodb/2.0.9```\n\nNote that you must be using MongoDB replica sets since this river tails the oplog.\n\n| MongoDB River Plugin     | ElasticSearch    | MongoDB       | TokuMX        |\n|--------------------------|------------------|---------------|---------------|\n| master                   | 1.7.3            | 3.0.0         | 1.5.1         |\n| 2.0.9                    | 1.4.2            | 3.0.0         | 1.5.1         |\n| 2.0.5                    | 1.4.2            | 2.6.6         | 1.5.1         |\n| 2.0.2                    | 1.3.5            | 2.6.5         | 1.5.1         |\n| 2.0.1                    | 1.2.2            | 2.4.9 -\u003e 2.6.3| 1.5.0         |\n| 2.0.0                    | 1.0.0 -\u003e 1.1.1   | 2.4.9         |               |\n| 1.7.4                    | 0.90.10          | 2.4.8         |               |\n| 1.7.3                    | 0.90.7           | 2.4.8         |               |\n| 1.7.2                    | 0.90.5           | 2.4.8         |               |\n| 1.7.1                    | 0.90.5           | 2.4.6         |               |\n| 1.7.0                    | 0.90.3           | 2.4.5         |               |\n| 1.6.11                   | 0.90.2           | 2.4.5         |               |\n| 1.6.9                    | 0.90.1           | 2.4.4         |               |\n| 1.6.8                    | 0.90.0           | 2.4.3         |               |\n| 1.6.7                    | 0.90.0           | 2.4.3         |               |\n| 1.6.6                    | 0.90.0           | 2.4.3         |               |\n\nBuild status\n-------\n\n[![Build Status](https://drone.io/github.com/richardwilly98/elasticsearch-river-mongodb/status.png)](https://drone.io/github.com/richardwilly98/elasticsearch-river-mongodb/latest)\n\n[Initial implementation](http://www.matt-reid.co.uk/blog_post.php?id=68#\u0026slider1=4) by [aparo](https://github.com/aparo).\n\nModified to get the same structure as the other Elasticsearch rivers (like [CouchDB](http://www.elasticsearch.org/blog/2010/09/28/the_river_searchable_couchdb.html))\n\nThe latest version monitors the oplog capped collection and supports attachment (GridFS).\n\nConfigure the river using the definition described in the [wiki](https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki):  \n\n\n```bash\n  curl -XPUT 'http://localhost:9200/_river/mongodb/_meta' -d '{\n    \"type\": \"mongodb\", \n    \"mongodb\": { \n      \"db\": \"DATABASE_NAME\", \n      \"collection\": \"COLLECTION\", \n      \"gridfs\": true\n    }, \n    \"index\": { \n      \"name\": \"ES_INDEX_NAME\", \n      \"type\": \"ES_TYPE_NAME\" \n    }\n  }'\n```\n\nExample:\n\n```bash\n  curl -XPUT 'http://localhost:9200/_river/mongodb/_meta' -d '{ \n    \"type\": \"mongodb\", \n    \"mongodb\": { \n      \"db\": \"testmongo\", \n      \"collection\": \"person\"\n    }, \n    \"index\": {\n      \"name\": \"mongoindex\", \n      \"type\": \"person\" \n    }\n  }'\n```\n\nImport data from mongo console:\n\n```bash\n  use testmongo\n  var p = {firstName: \"John\", lastName: \"Doe\"}\n  db.person.save(p)\n```\n\nQuery index:\n\n```bash\n  curl -XGET 'http://localhost:9200/mongoindex/_search?q=firstName:John'\n```\n\n```bash\n  curl -XPUT 'http://localhost:9200/_river/mongodb/_meta' -d '{ \n    \"type\": \"mongodb\", \n    \"mongodb\": { \n      \"db\": \"testmongo\", \n      \"collection\": \"fs\", \n      \"gridfs\": true \n    }, \n    \"index\": {\n      \"name\": \"mongoindex\", \n      \"type\": \"files\" \n    }\n  }'\n```\n\nImport binary content in mongo:\n\n```\n  %MONGO_HOME%\\bin\u003emongofiles.exe --host localhost:27017 --db testmongo --collection fs put test-document-2.pdf\n  connected to: localhost:27017\n  added file: { _id: ObjectId('4f230588a7da6e94984d88a1'), filename: \"test-document-2.pdf\", chunkSize: 262144, uploadDate: new Date(1327695240206), md5: \"c2f251205576566826f86cd969158f24\", length: 173293 }\n  done!\n```\n\nQuery index:\n\n```bash\n  curl -XGET 'http://localhost:9200/files/4f230588a7da6e94984d88a1?pretty=true'\n```\n\nAdmin URL: `http://localhost:9200/_plugin/river-mongodb/`\n\nSee more details check the [wiki](https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki)\n\nLicense\n-------\n\n    This software is licensed under the Apache 2 license, quoted below.\n\n    Copyright 2009-2012 Shay Banon and ElasticSearch \u003chttp://www.elasticsearch.org\u003e\n\n    Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n    use this file except in compliance with the License. You may obtain a copy of\n    the License at\n\n        http://www.apache.org/licenses/LICENSE-2.0\n\n    Unless required by applicable law or agreed to in writing, software\n    distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT\n    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the\n    License for the specific language governing permissions and limitations under\n    the License.\n\nChangelog\n-------\n\n#### 2.0.9\n- Update versions ES 1.4.2, MongoDB 3.0.0, MongoDB driver 2.13.0\n\n#### 2.0.5\n- Update versions ES 1.4.0\n- Bug fix for initial import of sharded collections\n- Bug fix for competing with other rivers for resources\n- Bug fix and test for skipping sync of elements when restarting river\n\n#### 2.0.2\n- Update versions ES 1.3.5, MongoDB 2.6.5, MongoDB driver 2.12.4, Java 7\n- Improved scaling for running multiple rivers and numerous bug fixes by @ankon of Collaborne B.V.\n- Improved TokuMX support by @kdkeck of Connectifier, Inc.\n- Support for collection names with period by @qraynaud\n\n#### 2.0.1\n- Update versions ES 1.2.2, MongoDB 2.6.3, MongoDB driver 2.12.3\n- Support for TokuMX by @kdkeck of Connectifier, Inc.\n\n#### 2.0.0\n- Update versions ES 1.0.0, MongoDB 2.4.9, MongoDB driver 2.11.4\n- Detection of stale river\n\n#### 1.7.4\n- Make include_fields work for nested fields\n- Fix ensuring river status indicates failures\n\n#### 1.7.3\n- Update versions ES 0.90.7\n- Optimization of oplog.rs query. The current query was too complex and not efficient in MongoDB. oplog.rs query now uses only $ts filter.\n- New ```options/import_all_collections``` parameter can be used to import all collection of a database (see issue [#177](https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/177))\n- Version, commit data and commit id are displayed in the log when the river starts.\n- New ```options/store_statistics``` parameter can be used to store statistic each time bulk processor is flushed. Data are store in _river/{river.name}\n- Default value for ```index/bulk/concurrent_bulk_requests``` has been changed to the number of cores available.\n- Capture failures from bulk processor and set status to IMPORT_FAILED when found.\n- Fix issues in document indexed counter in administration.\n- Refactoring to use 1 bulk processor per index/type.\n\n#### 1.7.2\n- Update versions MongoDB 2.4.8\n- Optimization of oplog.rs filter (see issue [#123](https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/123))\n- ```options/drop_collection``` will also track ```dropDatabase``` (see issue [#133](https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/133))\n- Add filter in the initial import by @bernd (see issue [#157](https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/157))\n- MongoDB default port is 27017 (if not specific in configuration) (see issue [#159](https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/159))\n- Allow alias in place of index (see issue [#163](https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/163))\n- Initial import supports existing index (see issue [#167](https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/167))\n- New parameter ```options/skip_initial_import``` to skip initial import using collection data. Default value is false.\n- Stop properly the river when deleted (see issue [#169](https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/169))\n- Administration updates: stopped river can be deleted from the administration, auto-refresh.\n- Switch to BulkProcessor API to provide more flexible configuration during bulk indexing. (see [commit](https://github.com/richardwilly98/elasticsearch-river-mongodb/commit/973dba973f6c1c353a38ed060754967504485769))\n\n#### 1.7.1\n- Update versions ES 0.90.5, MongoDB driver 2.11.3\n- Initial import using the collection by @benmccann. (see issue [#47](https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/47))\n- Add unit test to validate Chinese support (see issue [#95](https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/95))\n- Ensure MongoDB cursor is closed (see issue [#comment-24427369](https://github.com/richardwilly98/elasticsearch-river-mongodb/pull/106#issuecomment-24427369))\n- River administration has been improved. (see issue [#109](https://github.com/richardwilly98/elasticsearch-river-mongodb/pull/109))\n- Allow fields ```ts``` or ```op``` to be used in user collection. (see pr [#136](https://github.com/richardwilly98/elasticsearch-river-mongodb/pull/136))\n- Use of OPLOG_REPLAY to query oplog.rs\n\n#### 1.7.0\n- Update versions ES 0.90.3, MongoDB 2.4.6\n- Ability to index documents from a given datetime (see issue [#102](https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/102))\n- Fix for ```options/exclude_fields``` by @ozanozen (see issue [#103](https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/103))\n- Fix for ```options/drop_collection``` (see issue [#105](https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/105))\n- New advanced transformation feature.  (see issue [#106](https://github.com/richardwilly98/elasticsearch-river-mongodb/pull/106))\n- Add site to the river. Initial implementation (only start / stop river or display river settings).  (see issue [#109](https://github.com/richardwilly98/elasticsearch-river-mongodb/pull/109))\n- Implement include fields (see issue [#119](https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/119))\n- Refactoring of the river definition (new class MongoDBRiverDefinition).\n\n#### 1.6.11\n- Add SSL support by @alistair (see [#94](https://github.com/richardwilly98/elasticsearch-river-mongodb/pull/94))\n- Add support for $set operation (see issue [#91](https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/91))\n- Add Groovy unit test (for feature [#87](https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/87))\n- Update versions ES 0.90.2, MongoDB 2.4.5 and MongoDB driver 2.11.2\n- Fix for ```options/drop_collection``` option (issue [#79](https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/79))\n- New ```options/include_collection``` parameter to include the collection name in the document indexed. (see [#101](https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/101))\n\n#### 1.6.9\n- Allow the script filters to modify the document id (see [#83](https://github.com/richardwilly98/elasticsearch-river-mongodb/pull/83))\n- Support for Elasticsearch 0.90.1 and MongoDB 2.4.4\n- Improve exclude fields (support multi-level - see [#76](https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/76))\n- Fix to support ObjectId (see issue [#85](https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/85))\n- Add logger object to script filters\n- Provide example for Groovy script (see issue[#87](https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/87))\n\n#### 1.6.8 \n- Implement exclude fields (see issue #76)\n- Improve reconnection to MongoDB when connection is lost (see issue #77)\n- Implement drop collection feature (see issue #79). The river will drop all documents from the index type.\n\n#### 1.6.7 \n- Issue with sharded collection (see issue #46)\n\n#### 1.6.6 \n- Support for Elasticsearch 0.90.0 and MongoDB 2.4.3\n- MongoDB driver 2.11.1 (use of MongoClient)  \n\n\nBuilding from master\n-------\n\nInstall Maven (e.g `brew install maven`). Be sure `mvn` command is available.\n\nClone the repository:\n\n    git clone https://github.com/richardwilly98/elasticsearch-river-mongodb.git\n\nExecute installation script (inside the dir where you cloned the project):\n\n    ./elasticsearch-river-mongodb/install-local.sh\n\nIf your elasticsearch is not installed in `/usr/share/elasticsearch` you can set it `ES_HOME`. eg:\n\n    ES_HOME=/usr/local/Cellar/elasticsearch/1.1.1/ ./elasticsearch-river-mongodb/install-local.sh\n\n\nDeveloping in Eclipse\n---------------------\n\nInstall the m2eclipse plugin:\n* Work with: --All Available Sites--\n* Under Collaboration choose the m2e plugins\n\n[Install the testng eclipse plugin](http://testng.org/doc/download.html)\n\nRun `mvn eclipse:eclipse -DdownloadSources=true`\n\nImport the project with `File \u003e Import \u003e Maven \u003e Existing Maven Projects`\n\n\n[![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/richardwilly98/elasticsearch-river-mongodb/trend.png)](https://bitdeli.com/free \"Bitdeli Badge\")\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frichardwilly98%2Felasticsearch-river-mongodb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frichardwilly98%2Felasticsearch-river-mongodb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frichardwilly98%2Felasticsearch-river-mongodb/lists"}