Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/hckhanh/pg2bigquery

A CLI tool to convert query from PostgreSQL to BigQuery
https://github.com/hckhanh/pg2bigquery

big bigquery google pg pgsql postgres postgres-tool postgresql postgresql-database postgressql query query-parser querybuilder sql sql-toolkit sql-tools tool toolbox toolkit utility

Last synced: 8 days ago
JSON representation

A CLI tool to convert query from PostgreSQL to BigQuery

Host: GitHub
URL: https://github.com/hckhanh/pg2bigquery
Owner: hckhanh
Created: 2020-12-08T02:55:06.000Z (about 4 years ago)
Default Branch: main
Last Pushed: 2023-01-24T17:03:57.000Z (about 2 years ago)
Last Synced: 2025-01-03T17:20:56.071Z (about 1 month ago)
Topics: big, bigquery, google, pg, pgsql, postgres, postgres-tool, postgresql, postgresql-database, postgressql, query, query-parser, querybuilder, sql, sql-toolkit, sql-tools, tool, toolbox, toolkit, utility
Language: TypeScript
Homepage:
Size: 1.53 MB
Stars: 0
Watchers: 2
Forks: 0
Open Issues: 43
Metadata Files:
- Readme: README.md

Awesome Lists containing this project

README

        # pg2bigquery

A CLI tool to convert query from PostgreSQL to BigQuery

[![deepcode](https://www.deepcode.ai/api/gh/badge?key=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJwbGF0Zm9ybTEiOiJnaCIsIm93bmVyMSI6Imhja2hhbmgiLCJyZXBvMSI6InBnMmJpZ3F1ZXJ5IiwiaW5jbHVkZUxpbnQiOmZhbHNlLCJhdXRob3JJZCI6Mjg0MzQsImlhdCI6MTYxNzQ5NzY2MX0.RKs5KwUX_1aAQz-K1LyWD7HvjGi3zcIHhb-CVdZckE4)](https://www.deepcode.ai/app/gh/hckhanh/pg2bigquery/_/dashboard?utm_content=gh%2Fhckhanh%2Fpg2bigquery)

![Release](https://github.com/hckhanh/pg2bigquery/workflows/Release/badge.svg)

[![Language grade: JavaScript](https://img.shields.io/lgtm/grade/javascript/g/hckhanh/pg2bigquery.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/hckhanh/pg2bigquery/context:javascript)

[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)

[![Version](https://img.shields.io/npm/v/pg2bigquery.svg)](https://npmjs.org/package/pg2bigquery)

[![Downloads/week](https://img.shields.io/npm/dw/pg2bigquery.svg)](https://npmjs.org/package/pg2bigquery)

[![License](https://img.shields.io/npm/l/pg2bigquery.svg)](https://github.com/hckhanh/pg2bigquery/blob/master/package.json)

# Installation

For npm:

```shell

npm i -g pg2bigquery

```

or yarn:

```shell

yarn global add pg2bigquery

```

# Usage

## Requirements

```sh-session

$ pg2bigquery -d dataset -t tables.json input output

```

You need these things to use:

- **input**: input folder that contains PostgresSQL query files

- **output**: output folder that contains BigQuery query files

- **dataset**: destination dataset which is used to run BigQuery query

- **tables**: list of tables of pg database in json file

> If you want to know about the files format, go to [samples](samples)

### Tables JSON file

For now, you need to put the list of tables in a json file like this:

```json

["forms", "roles", "steps", "wards", "people", "staffs", "products"]

```

You can easily get the list of tables in your sql editor by this query:

```sql

select table_name from information_schema.tables;

```

Run `pg2bigquery` with `--help` option to get more details:

```sh-session

$ pg2bigquery --help

```

A success run will be:

```sh-session

$ pg2bigquery -d dataset -t tables.json input output

check output folder... done

get input files... 10 files

get tables... 55 tables

convert 10 files... done

```

# Limitations

> [No Silver Bullet](https://en.wikipedia.org/wiki/No_Silver_Bullet).

>

> The converted files might not be perfect.

When you use my tool, you will see these cases:

### `No matching signature for operator` (two different types)

```

No matching signature for operator = for argument types: INT64, STRING. Supported signature: ANY = ANY at [382:44]

```

BigQuery don't implicit cast one type to another when using with operators (+, -, <, >,...)

so you have to do it for yourself.