https://github.com/kvendingoldo/habr-bq-to-ch
https://github.com/kvendingoldo/habr-bq-to-ch
Last synced: 3 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/kvendingoldo/habr-bq-to-ch
- Owner: kvendingoldo
- Created: 2024-09-03T01:56:38.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2024-09-10T11:42:33.000Z (9 months ago)
- Last Synced: 2025-03-28T03:30:17.609Z (3 months ago)
- Language: HCL
- Size: 10.7 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# habr-bq-to-ch
Habr article about the migration from BigQuery to Clickhouse is available by the [link](https://habr.com/ru/companies/merkteam/articles/841900/).
## Export data from BigQuery to Yandex S3
```shell
export GOOGLE_APPLICATION_CREDENTIALS=""
export BOTO_CONFIG="./credentials.boto"
python3 00_migrate.py \
--bq_project= \
--bq_location= \
--gs_bucket= \
--yc_bucket=
```
## Import data from Yandex S3 to ClickHouse
```shell
export GOOGLE_APPLICATION_CREDENTIALS=""
export BOTO_CONFIG="./credentials.boto"
python3 01_import.py \
--s3_bucket_name="" \
--s3_access_key='' \
--s3_secret_key='' \
--ch_cluster_id='' \
--ch_host='clickhouse_cluster_host' \
--ch_password='' \
--ch_database='clickhouse_database' \
--bq_table_pattern='' \
--bq_project_id="" \
--bq_database=""
```