{"id":13399176,"url":"https://github.com/stripe-archive/mosql","last_synced_at":"2025-09-28T23:32:12.151Z","repository":{"id":6376032,"uuid":"7613509","full_name":"stripe-archive/mosql","owner":"stripe-archive","description":"MongoDB → PostgreSQL streaming replication        ","archived":true,"fork":false,"pushed_at":"2020-04-07T20:36:11.000Z","size":260,"stargazers_count":1630,"open_issues_count":58,"forks_count":226,"subscribers_count":84,"default_branch":"master","last_synced_at":"2024-04-13T18:21:21.637Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stripe-archive.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-01-14T21:57:04.000Z","updated_at":"2024-04-13T02:46:24.000Z","dependencies_parsed_at":"2022-07-14T07:20:36.521Z","dependency_job_id":null,"html_url":"https://github.com/stripe-archive/mosql","commit_stats":null,"previous_names":["stripe/mosql"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stripe-archive%2Fmosql","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stripe-archive%2Fmosql/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stripe-archive%2Fmosql/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stripe-archive%2Fmosql/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stripe-archive","download_url":"https://codeload.github.com/stripe-archive/mosql/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234575214,"owners_count":18854924,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T19:00:34.893Z","updated_at":"2025-09-28T23:32:11.812Z","avatar_url":"https://github.com/stripe-archive.png","language":"Ruby","funding_links":[],"categories":["Ruby"],"sub_categories":[],"readme":"# MoSQL: a MongoDB → SQL streaming translator\n\n\u003e _**MoSQL is no longer being actively maintained.**_\n\u003e _If you are interested in helping maintain this repository, please let us know.  We would love for it to find a forever home with someone who can give it the love it needs!_\n\nAt Stripe, we love MongoDB. We love the flexibility it gives us in\nchanging data schemas as we grow and learn, and we love its\noperational properties. We love replsets. We love the uniform query\nlanguage that doesn't require generating and parsing strings, tracking\nplaceholder parameters, or any of that nonsense.\n\nThe thing is, we also love SQL. We love the ease of doing ad-hoc data\nanalysis over small-to-mid-size datasets in SQL. We love doing JOINs\nto pull together reports summarizing properties across multiple\ndatasets. We love the fact that virtually every employee we hire\nalready knows SQL and is comfortable using it to ask and answer\nquestions about data.\n\nSo, we thought, why can't we have the best of both worlds? Thus:\nMoSQL.\n\n# MoSQL: Put Mo' SQL in your NoSQL\n\n![MoSQL](https://stripe.com/img/blog/posts/mosql/mosql.png)\n\nMoSQL imports the contents of your MongoDB database cluster into a\nPostgreSQL instance, using an oplog tailer to keep the SQL mirror live\nup-to-date. This lets you run production services against a MongoDB\ndatabase, and then run offline analytics or reporting using the full\npower of SQL.\n\n## Installation\n\nInstall from Rubygems as:\n\n    $ gem install mosql\n\nOr build from source by:\n\n    $ gem build mosql.gemspec\n\nAnd then install the built gem.\n\n## The Collection Map file\n\nIn order to define a SQL schema and import your data, MoSQL needs a\ncollection map file describing the schema of your MongoDB data. (Don't\nworry -- MoSQL can handle it if your mongo data doesn't always exactly\nfit the stated schema. More on that later).\n\nThe collection map is a YAML file describing the databases and\ncollections in Mongo that you want to import, in terms of their SQL\ntypes. An example collection map might be:\n\n\n    mongodb:\n      blog_posts:\n        :columns:\n        - id:\n          :source: _id\n          :type: TEXT\n        - author_name:\n          :source: author.name\n          :type: TEXT\n        - author_bio:\n          :source: author.bio\n          :type: TEXT\n        - title: TEXT\n        - created: DOUBLE PRECISION\n        :meta:\n          :table: blog_posts\n          :extra_props: true\n\nSaid another way, the collection map is a YAML file containing a hash\nmapping\n\n    \u003cMongo DB name\u003e -\u003e { \u003cMongo Collection Name\u003e -\u003e \u003cCollection Definition\u003e }\n\nWhere a `\u003cCollection Definition\u003e` is a hash with `:columns` and\n`:meta` fields.\n\n`:columns` is a list of hashes mapping SQL column names to an hash\ndescribing that column. This hash may contain the following fields:\n\n  * `:source`: The name of the attribute inside of MongoDB.\n  * `:type`: (Mandatory) The SQL type.\n\n\nUse of the `:source` attribute allows for renaming attributes, and\nextracting elements of a nested hash using MongoDB's\n[dot notation][dot-notation]. In the above example, the `name` and\n`bio` fields of the `author` sub-document will be expanded, and the\nMongoDB `_id` field will be mapped to an SQL `id` column.\n\nAt present, MoSQL does not support using the dot notation to access\nelements inside arrays.\n\nAs a shorthand, you can specify a one-element hash of the form `name:\nTYPE`, in which case `name` will be used for both the source attribute\nand the name of the destination column. You can see this shorthand for\nthe `title` and `created` attributes, above.\n\nEvery defined collection must include a mapping for the `_id`\nattribute.\n\n`:meta` contains metadata about this collection/table. It is\nrequired to include at least `:table`, naming the SQL table this\ncollection will be mapped to. `extra_props` determines the handling of\nunknown fields in MongoDB objects -- more about that later.\n\nBy default, `mosql` looks for a collection map in a file named\n`collections.yml` in your current working directory, but you can\nspecify a different one with `-c` or `--collections`.\n\n[dot-notation]: http://docs.mongodb.org/manual/core/document/#dot-notation\n\n## Usage\n\nOnce you have a collection map. MoSQL usage is easy. The basic form\nis:\n\n    mosql [-c collections.yml] [--sql postgres://sql-server/sql-db] [--mongo mongodb://mongo-uri]\n\nBy default, `mosql` connects to both PostgreSQL and MongoDB instances\nrunning on default ports on localhost without authentication. You can\npoint it at different targets using the `--sql` and `--mongo`\ncommand-line parameters.\n\n`mosql` will:\n\n 1. Create the appropriate SQL tables\n 2. Import data from the Mongo database\n 3. Start tailing the mongo oplog, propagating changes from MongoDB to SQL.\n\n\nAfter the first run, `mosql` will store the status of the optailer in\nthe `mongo_sql` table in your SQL database, and automatically resume\nwhere it left off. `mosql` uses the replset name to keep track of\nwhich mongo database it's tailing, so that you can tail multiple\ndatabases into the same SQL database. If you want to tail the same\nreplSet, or multiple replSets with the same name, for some reason, you\ncan use the `--service` flag to change the name `mosql` uses to track\nstate.\n\nYou likely want to run `mosql` against a secondary node, at least for\nthe initial import, which will cause large amounts of disk activity on\nthe target node. One option is to specify this in your connect URI:\n\n    mosql --mongo mongodb://node1,node2,node3?readPreference=secondary\n\n(Note that this requires you be using at least version 1.8.3 of\n`mongo-ruby-driver`)\n\n## Advanced usage\n\nFor advanced scenarios, you can pass options to control mosql's\nbehavior. If you pass `--skip-tail`, mosql will do the initial import,\nbut not tail the oplog. This could be used, for example, to do an\nimport off of a backup snapshot, and then start the tailer on the live\ncluster. This can also be useful for hosted services where you do not\nhave access to the oplog.\n\nIf you need to force a fresh reimport, run `--reimport`, which will\ncause `mosql` to drop tables, create them anew, and do another import.\n\nNormaly, MoSQL will scan through a list of the databases on the mongo\nserver you connect to. You avoid this behavior by specifiying a specific\nmongo db to connect to with the `--only-db [dbname]` option. This is\nuseful for hosted services which do not let you list all databases (via\nthe `listDatabases` command).\n\n## Schema mismatches and _extra_props\n\nIf MoSQL encounters values in the MongoDB database that don't fit\nwithin the stated schema (e.g. a floating-point value in a INTEGER\nfield), it will log a warning, ignore the entire object, and continue.\n\nIf it encounters a MongoDB object with fields not listed in the\ncollection map, it will discard the extra fields, unless\n`:extra_props` is set in the `:meta` hash. If it is, it will collect\nany missing fields, JSON-encode them in a hash, and store the\nresulting text in `_extra_props` in SQL. You can set `:extra_props`\nto use `JSON`, `JSONB`, or `TEXT`.\n\nAs of PostgreSQL 9.3, you can declare columns as type \"JSON\" and use\nthe [native JSON support][pg-json] to inspect inside of JSON-encoded\ntypes. In earlier versions, you can write code in an extension\nlanguage, such as [plv8][plv8].\n\n[pg-json]: http://www.postgresql.org/docs/9.3/static/functions-json.html\n\n## Non-scalar types\n\nMoSQL supports array types, using the `INTEGER ARRAY` array type\nsyntax. This will cause MoSQL to create the column as an array type in\nPostgreSQL, and insert rows appropriately-formatted.\n\nFields with hash values, or array values that are not in an\nARRAY-typed column, will be transformed into JSON TEXT strings before\nbeing inserted into PostgreSQL.\n\n[plv8]: http://code.google.com/p/plv8js/\n\n## Authentication\n\nAt present, in order to use MoSQL with a MongoDB instance requiring\nauthentication, you must:\n\n- Have a user with access to the admin database.\n- Specify the `admin` database in the `--mongo` argument\n- Specify the username and password in the `--mongo` argument\n\ne.g.\n\n```\nmosql --mongo mongodb://$USER:$PASSWORD@$HOST/admin\n```\n\nIn order to use MongoDB 2.4's \"roles\" support (which is different from that in\n2.6), you need to create the user in the admin database, give it explicit read\naccess to the databases you want to copy *and* to the `local` database, and\nspecify authSource in the URL.  eg, connect to `mydb/admin` with the mongo shell\nand run:\n\n```\n\u003e db.addUser({user: \"replicator\", pwd: \"PASSWORD\", roles: [], otherDBRoles: {local: [\"read\"], sourceDb: [\"read\"]}})\n```\n\n(Note that `roles: []` ensures that this user has no special access to the\n`admin` database.)  Now specify:\n\n```\nmosql --mongo mongodb://$USER:$PASSWORD@$HOST/sourceDb?authSource=admin\n```\n\nI have not yet tested using MoSQL with 2.6's rewritten \"roles\" support. Drop me\na note if you figure out anything I should know.\n\n## Sharded clusters\n\nMoSQL does not have special support for sharded Mongo clusters at this\ntime. It should be possible to run a separate MoSQL instance against\neach of the individual backend shard replica sets, streaming into\nseparate PostgreSQL instances, but we have not actually tested this\nyet.\n\n# Development\n\nPatches and contributions are welcome! Please fork the project and\nopen a pull request on [github][github], or just report issues.\n\nMoSQL includes a small but hopefully-growing test suite. It assumes a\nrunning PostgreSQL and MongoDB instance on the local host. To run the\ntest suite, first install all of MoSQL's dependencies:\n```shell\nbundle install\n```\nThen, run the tests:\n```shell\nrake test\n```\nYou can also point the suite at a different target via environment\nvariables; See `test/functional/_lib.rb` for more information.\n\n[github]: https://github.com/stripe/mosql\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstripe-archive%2Fmosql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstripe-archive%2Fmosql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstripe-archive%2Fmosql/lists"}