{"id":27854124,"url":"https://github.com/uw-labs/mobiq","last_synced_at":"2026-05-08T01:41:42.225Z","repository":{"id":30487196,"uuid":"124909913","full_name":"uw-labs/mobiq","owner":"uw-labs","description":"Mongo -\u003e BigQuery importer","archived":false,"fork":false,"pushed_at":"2022-01-22T09:21:16.000Z","size":163,"stargazers_count":1,"open_issues_count":5,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-04T08:58:42.039Z","etag":null,"topics":["bigquery","looker","mongo"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/uw-labs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-03-12T15:24:29.000Z","updated_at":"2020-09-23T13:22:14.000Z","dependencies_parsed_at":"2022-08-07T15:15:57.164Z","dependency_job_id":null,"html_url":"https://github.com/uw-labs/mobiq","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uw-labs%2Fmobiq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uw-labs%2Fmobiq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uw-labs%2Fmobiq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uw-labs%2Fmobiq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/uw-labs","download_url":"https://codeload.github.com/uw-labs/mobiq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252310963,"owners_count":21727516,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","looker","mongo"],"created_at":"2025-05-04T08:58:45.788Z","updated_at":"2026-05-08T01:41:37.196Z","avatar_url":"https://github.com/uw-labs.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mobiq\n\nMobiq is a tool for importing mongo collections into google's big query, it supports automatic schema recognition and allows for customer transformations to be plugged in at run time.\n\nIt dumps data from mongo using cursor stream api, uploads it to gc storage bucket and runs a big query import from there.\n\n## installation\n\n```\nnpm i -g --save mobiq\n```\n\n## usage\n\n```\n  Usage: mobiq [options]\n\n  Options:\n\n    -V, --version                     output the version number\n    -d, --db \u003cdsn\u003e                    mongo DB DSN\n    -c, --db-collection \u003ccollection\u003e  collection to dump\n    --db-query [query]                query for .find()\n    --db-limit [limit]                record limit\n    --db-skip [skip]                  number of records to skip (default: 0)\n    --db-batch [batch]                cursor batch size (default: 1000)\n    -p, --bq-project \u003cproject\u003e        big query project name\n    -s, --bq-dataset \u003cdataset\u003e        big query data set\n    -t, --bq-table [table]            big query table, defaults to --db-collection (default: )\n    -b, --bq-bucket \u003cbucket\u003e          gc bucket to use for data transfer\n    -k, --bq-credentials \u003cfile\u003e       gc credentials file (default: )\n    --transform-flatten-objects       enables flattening of nested hashes into single dimension ones\n    --no-transform-remove-nulls       disables removal of null value keys\n    --transform [file]                add transformation stream from file (default: )\n    --bq-option [option]              add big query import option key=value (default: [object Object])\n    --dump-schema [file]              do not import, just dump schema\n    --schema [file]                   use schema from file rather than guessing from files\n    --ls-transform                    list available transformations\n    -h, --help                        output usage information\n\n\n```\n\n## using transformations\n\nyou can list available transformations with `mobiq --ls-transform`, you can also provide your own and point at the relevant file via `--transform` switch.\n\nA transformation is a node transform stream running in object mode which gets each record in a mongo connection.\n\nExample:\n\n```node.js\nconst { Transform } = require('stream');\n\nmodule.exports = class TimestampFromMongoId extends Transform {\n    constructor() {\n        super({objectMode: true})\n    }\n\n    _transform(chunk, encoding, callback) {\n        chunk.createdOn = new Date(parseInt(chunk.id.substring(0, 8), 16) * 1000);\n        this.push(chunk);\n        callback()\n    }\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuw-labs%2Fmobiq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fuw-labs%2Fmobiq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuw-labs%2Fmobiq/lists"}