{"id":13462508,"url":"https://github.com/ozlerhakan/mongolastic","last_synced_at":"2025-04-12T06:30:38.812Z","repository":{"id":32322070,"uuid":"35897314","full_name":"ozlerhakan/mongolastic","owner":"ozlerhakan","description":":traffic_light: A dataset migration tool from MongoDB to Elasticsearch and vice versa. ","archived":true,"fork":false,"pushed_at":"2021-05-21T06:41:21.000Z","size":202,"stargazers_count":135,"open_issues_count":6,"forks_count":34,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-03-29T08:15:30.254Z","etag":null,"topics":["connector","converter","elasticsearch","java","map","migration","mongodb"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ozlerhakan.png","metadata":{"files":{"readme":"README.adoc","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-05-19T17:23:13.000Z","updated_at":"2025-01-09T22:43:31.000Z","dependencies_parsed_at":"2022-07-18T17:00:37.351Z","dependency_job_id":null,"html_url":"https://github.com/ozlerhakan/mongolastic","commit_stats":null,"previous_names":[],"tags_count":23,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ozlerhakan%2Fmongolastic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ozlerhakan%2Fmongolastic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ozlerhakan%2Fmongolastic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ozlerhakan%2Fmongolastic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ozlerhakan","download_url":"https://codeload.github.com/ozlerhakan/mongolastic/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248529154,"owners_count":21119453,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["connector","converter","elasticsearch","java","map","migration","mongodb"],"created_at":"2024-07-31T12:00:50.151Z","updated_at":"2025-04-12T06:30:37.902Z","avatar_url":"https://github.com/ozlerhakan.png","language":"Java","funding_links":[],"categories":["Uncategorized","Tools"],"sub_categories":["Uncategorized","Data"],"readme":"= Mongolastic\n:version: v1.4.3\n\nimage:https://travis-ci.org/ozlerhakan/mongolastic.svg?branch=master[\"Build Status\", link=\"https://travis-ci.org/ozlerhakan/mongolastic\"]\nimage:https://api.codacy.com/project/badge/Grade/8d768c2fc37246019115e4d090a33b98[\"Codacy code quality\", link=\"https://www.codacy.com/app/ozlerhakan/mongolastic?utm_source=github.com\u0026utm_medium=referral\u0026utm_content=ozlerhakan/mongolastic\u0026utm_campaign=Badge_Grade\"]\nimage:https://img.shields.io/docker/pulls/ozlerhakan/mongolastic.svg[\"Docker Pulls,link=\"https://hub.docker.com/r/ozlerhakan/mongolastic\"]\nimage:https://img.shields.io/github/release/ozlerhakan/mongolastic.svg[]\nimage:https://img.shields.io/badge/mongo.java.driver-3.4.2-brightgreen.svg[] image:https://img.shields.io/badge/elastic.java.driver-6.2.4-brightgreen.svg[]\nimage:https://img.shields.io/badge/license-MIT-blue.svg[]\n\nMongolastic enables you to migrate your datasets from a mongod node to an elasticsearch node and vice versa. Since mongo and elastic servers can run with different characteristics, the tool provides several optional and required features to ably connect them. Mongolastic works with a yaml or json configuration file to begin a migration process. It reads your demand on the file and start syncing data in the specified direction.\n\n== How it works\n\nFirst, you can either pull the corresponding image of the app from https://hub.docker.com/r/ozlerhakan/mongolastic/[Docker Hub]\n\nSupported tags and respective Dockerfile links:\n\n*  `_1.4_`, `_1.4.4_`, `_latest_` https://github.com/ozlerhakan/mongolastic/blob/master/Dockerfile[_(master/Dockerfile)_]\n*  `_1.4.3_` https://github.com/ozlerhakan/mongolastic/blob/0dacd80cbdf7b5b7b282bf6dd89ede8558021577/Dockerfile[_(1.4.2/Dockerfile)_]\n\n\nor download the latest https://github.com/ozlerhakan/mongolastic/releases/download/{version}/mongolastic.jar[mongolastic.jar] file.\n\nSecond, create a yaml or json file which must contain the following structure:\n\n[source,yaml]\n----\nmisc:\n    dindex:\n        name: \u003cstring\u003e      \u003c1\u003e\n        as: \u003cstring\u003e        \u003c2\u003e\n    ctype:\n        name: \u003cstring\u003e      \u003c3\u003e\n        as: \u003cstring\u003e        \u003c4\u003e\n    direction: (em | me)    \u003c5\u003e\n    batch: \u003cnumber\u003e         \u003c6\u003e\n    dropDataset: \u003cbool\u003e     \u003c7\u003e\nmongo:\n    host: \u003cip-address\u003e      \u003c8\u003e\n    port: \u003cnumber\u003e          \u003c9\u003e\n    query: \"mongo-query\"    \u003c10\u003e\n    project: \"projection\"   \u003c11\u003e\n    auth:                   \u003c12\u003e\n        user: \u003cstring\u003e\n        pwd: \"password\"\n        source: \u003cdb-name\u003e\n        mechanism: ( plain | scram-sha-1 | x509 | gssapi | cr )\nelastic:\n    host: \u003cip-address\u003e     \u003c13\u003e\n    port: \u003cnumber\u003e         \u003c14\u003e\n    dateFormat: \"\u003cformat\u003e\" \u003c15\u003e\n    longToString: \u003cbool\u003e   \u003c16\u003e\n    clusterName: \u003cstring\u003e  \u003c17\u003e\n    auth:                  \u003c18\u003e\n        user: \u003cstring\u003e\n        pwd: \"password\"\n----\n\u003c1\u003e  the _database/index name_ to connect to.\n\u003c2\u003e  another _database/index name_ in which documents will be located in the target service (*Optional*)\n\u003c3\u003e  the _collection/type name_ to export.\n\u003c4\u003e  another _collection/type name_ in which indexed/collected documents will reside in the target service (*Optional*)\n\u003c5\u003e  direction of the data transfer. the default direction is me (that is, mongo to elasticsearch). You can skip this option if your data move from mongo to es.\n\u003c6\u003e  Override the default batch size which is normally 200. (*Optional*)\n\u003c7\u003e  configures whether or not the target table should be dropped prior to loading data. Default value is true (*Optional*)\n\u003c8\u003e  the name of the host machine where the `mongod` is running.\n\u003c9\u003e  the port where the `mongod` instance is listening.\n\u003c10\u003e  data will be transferred based on a json mongodb query (*Optional*)\n\u003c11\u003e with 1.4.1, you can manipulate documents that will be migrated from mongo to es based on the https://docs.mongodb.com/manual/reference/operator/aggregation/project/[`$project`] operator (*Optional*)\n\u003c12\u003e as of v1.3.5, you can access an auth mongodb by giving auth configuration. (*Optional*)\n\u003c13\u003e the name of the host machine where the `elastic node` is running.\n\u003c14\u003e the *transport* port where the transport module will communicate with the running elastic node. E.g. *9300* for node-to-node communication.\n\u003c15\u003e a custom formatter for Date fields rather than the default DateCodec (*Optional*)\n\u003c16\u003e serialize long value as a string for backwards compatibility with other tools (*Optional*)\n\u003c17\u003e connect to a spesific elastic cluster (*Optional*)\n\u003c18\u003e as of v1.3.9, you can access an auth elastic search by giving auth configuration. (*Optional*)\n\n---\n\nAlternatively, a JSON file can be specified as a mongolastic configuration file including the same YAML file structure above.\n\n[source,json]\n----\n{\n\t\"misc\": {\n\t\t\"dindex\": {\n\t\t\t\"name\": \"twitter\",\n\t\t\t\"as\": \"media\"\n\t\t},\n\t\t\"ctype\": {\n\t\t\t\"name\": \"tweets\",\n\t\t\t\"as\": \"posts\"\n\t\t},\n\t\t\"direction\": \"me\",\n\t\t\"batch\": 400,\n\t\t\"dropDataset\": true\n\t},\n\t\"mongo\": {\n\t\t\"host\": \"127.0.0.1\",\n\t\t\"port\": 27017,\n\t\t\"query\": \"{ lang: 'en' }\",\n\t\t\"project\": \"{ user:1, name:'$user.name', location: { $substr: [ '$user.location', 10, 15 ] }}\",\n\t\t\"auth\": {\n\t\t\t\"user\": \"joe\",\n\t\t\t\"pwd\": \"1234\",\n\t\t\t\"source\": \"twitter\",\n\t\t\t\"mechanism\": \"scram-sha-1\"\n\t\t}\n\t},\n\t\"elastic\": {\n\t\t\"host\": \"127.0.0.1\",\n\t\t\"port\": 9300,\n\t\t\"dateFormat\": \"yyyy-MM-dd\",\n\t\t\"longToString\": true,\n\t\t\"auth\": {\n\t\t\t\"user\": \"joe\",\n\t\t\t\"pwd\": \"4321\"\n\t\t}\n\t}\n}\n----\n\n== Example #1\n\nThe following files have the same configuration details:\n\n.yaml file\n[source,yaml]\n----\nmisc:\n    dindex:\n        name: twitter\n        as: kodcu\n    ctype:\n        name: tweets\n        as: posts\nmongo:\n    host: localhost\n    port: 27017\n    query: \"{ 'user.name' : 'kodcu.com'}\"\nelastic:\n    host: localhost\n    port: 9300\n----\n\n.json file\n[source,json]\n----\n{\n\t\"misc\": {\n\t\t\"dindex\": {\n\t\t\t\"name\": \"twitter\",\n\t\t\t\"as\": \"kodcu\"\n\t\t},\n\t\t\"ctype\": {\n\t\t\t\"name\": \"tweets\",\n\t\t\t\"as\": \"posts\"\n\t\t}\n\t},\n\t\"mongo\": {\n\t\t\"host\": \"localhost\",\n\t\t\"port\": 27017,\n\t\t\"query\": \"{ 'user.name' : 'kodcu.com'}\"\n\t},\n\t\"elastic\": {\n\t\t\"host\": \"localhost\",\n\t\t\"port\": 9300\n\t}\n}\n----\n\nthe config says that the transfer direction is from mongodb to elasticsearch, mongolastic first looks at the _tweets_ collection, where the _user name_ is _kodcu.com_, of the _twitter_ database located on a mongod server running on default host interface and port number. If It finds the corresponding data, It will start copying those into an elasticsearch environment running on default host and transport number. After all, you should see a type called _\"posts\"_ in an index called _\"kodcu\"_ in the current elastic node. Why the index and type are different is because \"dindex.as\" and \"ctype.as\" options were set, these indicates that your data being transferred exist in _posts_ type of the _kodcu_ index.\n\nAfter downloading the jar or pulling the image and providing a conf file, you can either run the tool as:\n\n    $ java -jar mongolastic.jar -f config.file\n\n__or__\n\n    $ docker run --rm -v $(PWD)/config.file:/config.file --net host ozlerhakan/mongolastic:\u003ctag\u003e config.file\n\n== Example #2\n\nUsing the project field, you are able to manipulate documents when migrating them from mongodb to elasticsearch. For more examples about the `$project` operator of the aggregation pipeline, take a look at its https://docs.mongodb.com/manual/reference/operator/aggregation/project/[documentation].\n\n[source,yaml]\n----\nmisc:\n    dindex:\n        name: twitter\n    ctype:\n        name: tweets\nmongo:\n    host: 192.168.10.151\n    port: 27017\n    project: \"{ user: 1, name: '$user.name', location: { $substr: [ '$user.location', 10, 15 ] }}\" \u003c1\u003e\nelastic:\n    host: 192.168.10.152\n    port: 9300\n----\n\u003c1\u003e the migrated documents will include the user field and contain new fields `name` and `location`.\n\nNOTE: Every attempt of running the tool drops the mentioned db/index in the target environment unless the dropDataset parameter is configured otherwise.\n\n== License\n\nMongolastic is released under http://showalicense.com/?hide_explanations=false\u0026year=2015\u0026fullname=Kodcu.com#license-mit[MIT].\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fozlerhakan%2Fmongolastic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fozlerhakan%2Fmongolastic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fozlerhakan%2Fmongolastic/lists"}