{"id":41343707,"url":"https://github.com/mponce/google-cloud-dataflow-pipeline","last_synced_at":"2026-01-23T06:54:09.287Z","repository":{"id":225440731,"uuid":"130953240","full_name":"mponce/google-cloud-dataflow-pipeline","owner":"mponce","description":"Google Cloud DataFlow - Load CSV Files to BigQuery Tables","archived":false,"fork":false,"pushed_at":"2022-07-31T16:27:25.000Z","size":33,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-03-02T05:35:35.855Z","etag":null,"topics":["csv-import","google-bigquery","google-cloud-dataflow","google-cloud-storage"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mponce.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2018-04-25T05:04:42.000Z","updated_at":"2024-03-02T05:35:37.666Z","dependencies_parsed_at":"2024-03-02T05:45:56.932Z","dependency_job_id":null,"html_url":"https://github.com/mponce/google-cloud-dataflow-pipeline","commit_stats":null,"previous_names":["mponce/google-cloud-dataflow-pipeline"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mponce/google-cloud-dataflow-pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mponce%2Fgoogle-cloud-dataflow-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mponce%2Fgoogle-cloud-dataflow-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mponce%2Fgoogle-cloud-dataflow-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mponce%2Fgoogle-cloud-dataflow-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mponce","download_url":"https://codeload.github.com/mponce/google-cloud-dataflow-pipeline/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mponce%2Fgoogle-cloud-dataflow-pipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28682263,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-23T05:48:07.525Z","status":"ssl_error","status_checked_at":"2026-01-23T05:48:07.129Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv-import","google-bigquery","google-cloud-dataflow","google-cloud-storage"],"created_at":"2026-01-23T06:54:08.701Z","updated_at":"2026-01-23T06:54:09.275Z","avatar_url":"https://github.com/mponce.png","language":"Java","readme":"### Introduction\n\nThis is a demo on how to load data into Google BigQuery using Google Cloud [DataFlow](https://cloud.google.com/dataflow/) pipelines.\n\nGoogle Cloud DataFlow is an Apache Beam Pipeline [runner](https://beam.apache.org/documentation/runners/capability-matrix/). A Pipeline is a sequence of data transformations applied to collections of elements of different types.\n\n### Local SDK Setup\n\n\nDownload and install the Google Cloud SDK to your computer by running:\n\n```bash\nsudo curl https://sdk.cloud.google.com | bash\n```\nAfter installing the SDK you will have ```gcloud and gsutil``` commands to interact with GCP services.\n\nMake sure you login before compiling the project:\n```bash\ngcloud auth application-default login\n```\nThis will open the browser and redirect you the GCP services to authenticate your computer.\n\nOptionally set your default project:\n\n```bash\ngcloud config set project \u003cYOUR CLOUD PLATFORM PROJECT ID\u003e\n```\n\n### Test Data\n\nTo run this demo you need to have your GCP account setup. This code can run from the developer machine on the GCP or from local system with the GCP SDK installed. \n\nYou will also need to have some data in your Cloud Storage in CSV format to load to BigQuery. For SimplePipeline there is a sample ```products.csv``` in the ```data``` folder of this repo.  \n\nA sample project structure for data, staging and templates could be:\n\n**Storage Structure:**\n```text\n- \u003cPROJECT_ID\u003e/\n     +---/data/\n         +---/products.csv\n     +---/staging/ (If you remove this folder running the Template will fail)\n     +---/templates/\n         +---/SimplePipeline (this will be created by mvn compile command line)\n     +---/output/\n```\n\n### Compile and Run\n\nReplace the parameters below and run it from the command line:\n\n```bash\nmvn compile exec:java -Dexec.mainClass=com.example.SimplePipeline -Dexec.args=\"--project=\u003cPROJECT_ID\u003e \\\n    --jobName=\u003cJOB_NAME\u003e \\\n    --inputFile=gs://\u003cPROJECT_ID\u003e/csv/products.csv \\\n    --output=\u003cPROJECT_ID\u003e:\u003cBIGQUERY_DATASET_NAME\u003e.\u003cBIGQUERY_TABLE_NAME\u003e \\\n    --tempLocation=gs://\u003cPROJECT_ID\u003e/temp \\\n    --stagingLocation=gs://\u003cPROJECT_ID\u003e/staging \\\n    --runner=DataflowRunner\" \n```\n\nRun the ```mvn``` command again this time adding ```--templateLocation``` parameter to save the template in a storage location. The log will show a line with ```INFO: Template successfully created.``` You might get a NullPointerException error but it will still create the template. \n\n```bash\n    --templateLocation=gs://\u003cPROJECT_ID\u003e/templates/SimplePipeline \n```\n\n### Template Metadata\n\nUpload the file ```SimplePipeline_metadata``` to your Cloud Storage template location ```\u003cPROJECT_ID\u003e/templates``` (the same location specified by ```--templateLocation``` parameter) this will help the DataFlow UI automatically display the parameters required to run our SimplePipeline.\n\n### Run from DataFlow UI\n\nLogin to GCP console under **DataFlow** service create a new job from *Custom Template* and select **SimplePipeline** the template from the ```\u003cPROJECT_ID\u003e/templates``` folder. The Dataflow UI will read the **SimplePipeline_metadata** file (previously uploaded) to render the input parameters. Fill in those parameters and Run the job. \n\nIf all goes well you should be able to see the Job status (running, processed etc.) and options to see the Job *LOGS*.\n\n### Running as Scheduled Job\n\nOptionally create a web app to call the pipeline on scheduled basis. For this review ```DataPipeline``` class and update the GCP properties.\n\nTo deploy the servlet to App Engine run:\n\n```\nmvn appengine:deploy\n````\n\nModify the file ```cron.yaml``` to adjust the scheduling intervals to call the ```/schedule/url``` url that is mapped to the ```SchedulingServlet.java``` which runs ```DataPipeline``` class.\n\n\n\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmponce%2Fgoogle-cloud-dataflow-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmponce%2Fgoogle-cloud-dataflow-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmponce%2Fgoogle-cloud-dataflow-pipeline/lists"}