https://github.com/uw-labs/mobiq
Mongo -> BigQuery importer
https://github.com/uw-labs/mobiq
bigquery looker mongo
Last synced: about 1 month ago
JSON representation
Mongo -> BigQuery importer
- Host: GitHub
- URL: https://github.com/uw-labs/mobiq
- Owner: uw-labs
- License: lgpl-3.0
- Created: 2018-03-12T15:24:29.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2022-01-22T09:21:16.000Z (over 4 years ago)
- Last Synced: 2025-05-04T08:58:42.039Z (about 1 year ago)
- Topics: bigquery, looker, mongo
- Language: JavaScript
- Homepage:
- Size: 159 KB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# mobiq
Mobiq is a tool for importing mongo collections into google's big query, it supports automatic schema recognition and allows for customer transformations to be plugged in at run time.
It dumps data from mongo using cursor stream api, uploads it to gc storage bucket and runs a big query import from there.
## installation
```
npm i -g --save mobiq
```
## usage
```
Usage: mobiq [options]
Options:
-V, --version output the version number
-d, --db mongo DB DSN
-c, --db-collection collection to dump
--db-query [query] query for .find()
--db-limit [limit] record limit
--db-skip [skip] number of records to skip (default: 0)
--db-batch [batch] cursor batch size (default: 1000)
-p, --bq-project big query project name
-s, --bq-dataset big query data set
-t, --bq-table [table] big query table, defaults to --db-collection (default: )
-b, --bq-bucket gc bucket to use for data transfer
-k, --bq-credentials gc credentials file (default: )
--transform-flatten-objects enables flattening of nested hashes into single dimension ones
--no-transform-remove-nulls disables removal of null value keys
--transform [file] add transformation stream from file (default: )
--bq-option [option] add big query import option key=value (default: [object Object])
--dump-schema [file] do not import, just dump schema
--schema [file] use schema from file rather than guessing from files
--ls-transform list available transformations
-h, --help output usage information
```
## using transformations
you can list available transformations with `mobiq --ls-transform`, you can also provide your own and point at the relevant file via `--transform` switch.
A transformation is a node transform stream running in object mode which gets each record in a mongo connection.
Example:
```node.js
const { Transform } = require('stream');
module.exports = class TimestampFromMongoId extends Transform {
constructor() {
super({objectMode: true})
}
_transform(chunk, encoding, callback) {
chunk.createdOn = new Date(parseInt(chunk.id.substring(0, 8), 16) * 1000);
this.push(chunk);
callback()
}
}
```