{"id":13941566,"url":"https://github.com/victorcouste/google-cloudfunctions-dataprep","last_synced_at":"2025-07-20T04:31:53.562Z","repository":{"id":89953067,"uuid":"259056222","full_name":"victorcouste/google-cloudfunctions-dataprep","owner":"victorcouste","description":"Google Cloud Functions examples for Google Cloud Dataprep","archived":false,"fork":false,"pushed_at":"2021-02-12T15:54:50.000Z","size":294,"stargazers_count":11,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-11-27T10:36:11.714Z","etag":null,"topics":["api","api-rest","bigquery","cloud-functions","cloudfunctions-dataprep","dataprep","dataprep-job","google-bigquery","google-cloud-dataprep","google-sheet","trifacta"],"latest_commit_sha":null,"homepage":"https://cloud.google.com/dataprep","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/victorcouste.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-26T14:51:40.000Z","updated_at":"2022-12-07T20:00:40.000Z","dependencies_parsed_at":"2023-03-30T16:04:56.400Z","dependency_job_id":null,"html_url":"https://github.com/victorcouste/google-cloudfunctions-dataprep","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/victorcouste/google-cloudfunctions-dataprep","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victorcouste%2Fgoogle-cloudfunctions-dataprep","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victorcouste%2Fgoogle-cloudfunctions-dataprep/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victorcouste%2Fgoogle-cloudfunctions-dataprep/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victorcouste%2Fgoogle-cloudfunctions-dataprep/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/victorcouste","download_url":"https://codeload.github.com/victorcouste/google-cloudfunctions-dataprep/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victorcouste%2Fgoogle-cloudfunctions-dataprep/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266067262,"owners_count":23871324,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","api-rest","bigquery","cloud-functions","cloudfunctions-dataprep","dataprep","dataprep-job","google-bigquery","google-cloud-dataprep","google-sheet","trifacta"],"created_at":"2024-08-08T02:01:21.444Z","updated_at":"2025-07-20T04:31:53.551Z","avatar_url":"https://github.com/victorcouste.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"# Google Cloud Functions for Cloud Dataprep\n\n\u003cimg src=\"https://github.com/victorcouste/google-cloudfunctions-dataprep/raw/master/CloudFunctions_Dataprep.png\" width=\"70%\" height=\"70%\"\u003e\n\n[Google Cloud Functions](https://cloud.google.com/functions) examples for [Cloud Dataprep](https://cloud.google.com/dataprep)\n\n- **[gcs_trigger_dataprep_job.py](https://github.com/victorcouste/google-cloudfunctions-dataprep/blob/master/gcs_trigger_dataprep_job.py)** : Background Python function to trigger a Dataprep job when a file is created in a Google Cloud Storage bucket folder. Dataprep job started with REST API call and new file as parameter. Implementation details in the blog post [How to Automate a Cloud Dataprep Pipeline When a File Arrives](https://medium.com/google-cloud/how-to-automate-a-cloud-dataprep-pipeline-when-a-file-arrives-9b85f2745a09)\n\n- **[job-result-google-sheet.js](https://github.com/victorcouste/google-cloudfunctions-dataprep/blob/master/job-result-google-sheet.js)** : HTTP Node.js function to write in a Google Sheet a Dataprep job result info (id, status) with recipe name, link to the job page and link to PDF of result's profile. This HTTP Cloud function is called from a Dataprep Webhook when jobs are finished (success or failure). Implementation details in the blog post [Leverage Cloud Functions and APIs to Monitor Cloud Dataprep Jobs Status in a Google Sheet](https://towardsdatascience.com/leverage-cloud-functions-and-apis-to-monitor-cloud-dataprep-jobs-status-in-a-google-sheet-b412ee2b9acc).\n\n- **[publishing_googlesheet.js](https://github.com/victorcouste/google-cloudfunctions-dataprep/blob/master/publishing_googlesheet.js)** : HTTP Node.js function to publish Dataprep output in a Google Sheet. Google sheet name created will be based on the default single CSV file name generated in GSC + Dataprep Job id. In the Cloud Function code, you need to update your [Dataprep Token Access](https://docs.trifacta.com/display/DP/Access+Tokens+Page) (to call REST API) and the [Google Spreadsheet ID](https://developers.google.com/sheets/api/guides/concepts#spreadsheet_id). And this Cloud Function can be triggered when a Dataprep job is finished via a [Dataprep Webhook](https://docs.trifacta.com/display/DP/Create+Flow+Webhook+Task).\n\n- **[job-result-google-bigquery.py](https://github.com/victorcouste/google-cloudfunctions-dataprep/blob/master/job-result-google-bigquery.py)** : HTTP Python function to write in a Google BigQuery table a Dataprep job result info (id, status) with dataset output name (recipe name), Google user and link to the job page. This HTTP Cloud function is called from a Dataprep Webhook when jobs are finished (success or failure). Implementation details in the blog post [Monitor your BigQuery Data Warehouse Dataprep Pipeline with Data Studio](https://medium.com/google-cloud/monitor-your-bigquery-data-warehouse-dataprep-pipeline-with-data-studio-8e46b2beda1).\n\n- **[export_import_dataprep_flow.py](https://github.com/victorcouste/google-cloudfunctions-dataprep/blob/master/export_import_dataprep_flow.py)** : Export a Dataprep flow from a project and import it in another project. Option to save or get the flow package (zip file) in a GCS bucket folder.\n\n- **[Update Google Cloud Data Catalog](https://victorcouste.github.io/google-data-catalog-dataprep/)** : A Cloud Function to create or update Google Cloud Data Catalog tags on BigQuery tables with Cloud Dataprep Metadata and Column's Profile.\n\n\u003cbr/\u003e\u003cbr/\u003e\n\n\nGoogle Cloud Functions [https://cloud.google.com/functions](https://cloud.google.com/functions)\n\nCloud Dataprep by Trifacta [https://cloud.google.com/dataprep](https://cloud.google.com/dataprep)\n\nCloud Dataprep Standard API [https://api.trifacta.com/dataprep-standard](https://api.trifacta.com/dataprep-standard)\n\nCloud Dataprep Premium API [https://api.trifacta.com/dataprep-premium](https://api.trifacta.com/dataprep-premium)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvictorcouste%2Fgoogle-cloudfunctions-dataprep","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvictorcouste%2Fgoogle-cloudfunctions-dataprep","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvictorcouste%2Fgoogle-cloudfunctions-dataprep/lists"}