{"id":22385191,"url":"https://github.com/jasontanx/bigquery-to-postgres-etl","last_synced_at":"2026-05-15T22:02:02.866Z","repository":{"id":164059097,"uuid":"617996668","full_name":"jasontanx/bigquery-to-postgres-etl","owner":"jasontanx","description":"Data ingestion from Google BigQuery to PostgreSQL","archived":false,"fork":false,"pushed_at":"2023-04-05T12:45:06.000Z","size":19,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-01T02:19:54.233Z","etag":null,"topics":["big-query","data-engineering","etl-pipeline","postgresql"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jasontanx.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-23T14:42:04.000Z","updated_at":"2024-11-13T15:22:37.000Z","dependencies_parsed_at":null,"dependency_job_id":"b9669697-85fa-4763-a4e0-1e7d9cc22d37","html_url":"https://github.com/jasontanx/bigquery-to-postgres-etl","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jasontanx%2Fbigquery-to-postgres-etl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jasontanx%2Fbigquery-to-postgres-etl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jasontanx%2Fbigquery-to-postgres-etl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jasontanx%2Fbigquery-to-postgres-etl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jasontanx","download_url":"https://codeload.github.com/jasontanx/bigquery-to-postgres-etl/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245728740,"owners_count":20662744,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-query","data-engineering","etl-pipeline","postgresql"],"created_at":"2024-12-05T01:22:03.590Z","updated_at":"2025-10-31T19:05:00.946Z","avatar_url":"https://github.com/jasontanx.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# bigquery-to-postgres-etl\n\nETL --\u003e (Extract - Load)\n\n[ Source --\u003e BigQuery, Destination --\u003e Postgres]\n\n![image](https://user-images.githubusercontent.com/116934441/227949498-df847c06-6865-40bc-b434-59996e28de93.png)\n\n# Project Overview\n**Description**: Personal project (BQ to Postgres data ingestion)\n\n**Source**: GCP Big Query\n\n**Apache Airflow (Orchestration Tool)** : Not Involved in this project*\n\n**Destination**: Postgresql\n\n**Language**: python \n\n\n**Main Tools Involved:**\n1. Python (*IDE - Visual Studio Code*)\n2. Google Cloud Platform account (*For extracting data from BigQuery*)\n3. Docker (*Preferred*)\n\n\n**Workflow**\n1. Extract data from BigQuery via query\n2. Save queried output into dataframe and output data as csv format\n3. Ingest data into Postgres DB with new file name\n\n**Final Outcome**\n\n![image](https://user-images.githubusercontent.com/116934441/229813264-e66a18ec-b191-44b3-b3f0-ff384a73ce70.png)\n\n**Extra Information**\n\n1. Docker Compose tutorial (to run postgresql)\n\n--\u003e [DE Zoomcamp 1.2.5 - Running Postgres and pgAdmin with Docker-Compose](https://www.youtube.com/watch?v=hKI6PkPhpa0\u0026list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb\u0026index=9)\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjasontanx%2Fbigquery-to-postgres-etl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjasontanx%2Fbigquery-to-postgres-etl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjasontanx%2Fbigquery-to-postgres-etl/lists"}