{"id":24529468,"url":"https://github.com/sangnandar/csv-fixer-for-gcs","last_synced_at":"2025-06-16T14:07:38.666Z","repository":{"id":271777883,"uuid":"914533850","full_name":"sangnandar/CSV-Fixer-for-GCS","owner":"sangnandar","description":"Google Apps Script to read, validate, and fix CSV files stored in Google Cloud Storage (GCS).","archived":false,"fork":false,"pushed_at":"2025-02-22T12:45:36.000Z","size":26,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-15T17:47:07.616Z","etag":null,"topics":["csv-parsing","google-apps-script","google-cloud-storage"],"latest_commit_sha":null,"homepage":"https://script.google.com","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sangnandar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-09T19:32:37.000Z","updated_at":"2025-02-22T12:45:40.000Z","dependencies_parsed_at":"2025-01-09T20:41:31.987Z","dependency_job_id":null,"html_url":"https://github.com/sangnandar/CSV-Fixer-for-GCS","commit_stats":null,"previous_names":["sangnandar/csv-fixer-for-gcs"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sangnandar/CSV-Fixer-for-GCS","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sangnandar%2FCSV-Fixer-for-GCS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sangnandar%2FCSV-Fixer-for-GCS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sangnandar%2FCSV-Fixer-for-GCS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sangnandar%2FCSV-Fixer-for-GCS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sangnandar","download_url":"https://codeload.github.com/sangnandar/CSV-Fixer-for-GCS/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sangnandar%2FCSV-Fixer-for-GCS/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260173650,"owners_count":22969866,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv-parsing","google-apps-script","google-cloud-storage"],"created_at":"2025-01-22T07:37:35.372Z","updated_at":"2025-06-16T14:07:38.642Z","avatar_url":"https://github.com/sangnandar.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CSV Fixer for GCS\n\n## Overview\nThis project is a **Google Apps Script solution** designed to read, validate, and fix CSV files stored in **Google Cloud Storage (GCS)**.\n\nThis project extends the functionality of [Load CSVs from GCS to BigQuery](https://github.com/sangnandar/Load-CSVs-from-GCS-to-BigQuery). When errors are detected, CSV files cannot be loaded into BigQuery. This project addresses those errors, uploads the corrected files back to GCS, and prepares them for processing by \"Load CSVs from GCS to BigQuery\".\n\n### Features\n- **Error Detection:** Reads CSV files from GCS, identifies records with errors, and creates a new Google Sheet for each file, listing the issues found.\n- **Error Correction:** Enables users to fix the errors in Google Sheets and automatically uploads the corrected CSV files back to GCS.\n\n## How it works\n1. CSV files with errors are stored in the GCS bucket under the `csv/fail/` folder.\n2. The Apps Script:\n   - Reads the files.\n   - Extracts records with errors and writes them to a Google Sheet.\n3. Users review and fix the errors in the sheet.\n4. The fixed file is re-uploaded to the GCS `csv/` parent folder, making it ready for processing by \"Load CSVs from GCS to BigQuery\".\n\n## Installation\n\n### GCP Project configuration\nEnable the following API in your GCP project:\n- Cloud Storage API\n\n### Apps Script configuration\n- Set up Script Properties in **Apps Script -\u003e Project Settings -\u003e Script Properties**:\n   ```\n   {\n     BUCKET_NAME: \u003cGCS bucket name\u003e\n   }\n   ```\n\n- Configure the `appsscript.json` file:\n   ```json\n   {\n     \"oauthScopes\": [\n       \"https://www.googleapis.com/auth/spreadsheets\",\n       \"https://www.googleapis.com/auth/script.external_request\",\n       \"https://www.googleapis.com/auth/devstorage.read_only\"\n     ]\n   }\n   ```\n\n### Sheets configuration\n**DO NOT** change sheets name, delete columns, or re-arrange columns for the following ranges:\n- Read\n  ```\n  'Files'!A2:B\n  ```\n\nSheets layout\n\n![image](https://github.com/user-attachments/assets/6cf1e46a-18cd-4093-98ed-c8c2b7204502)\n\n## Usage\n1. Attach the Apps Script to a Google Sheets document.\n2. Use the custom menu in the toolbar:\n   - **Custom Menu -\u003e Check for errors:** reads the file and lists records with errors.\n   - **Custom Menu -\u003e Fix file:** uploads the corrected file back to GCS.\n\n# Related project\n- [Load CSVs from GCS to BigQuery](https://github.com/sangnandar/Load-CSVs-from-GCS-to-BigQuery) - Clean load CSV files from GCS to BigQuery.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsangnandar%2Fcsv-fixer-for-gcs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsangnandar%2Fcsv-fixer-for-gcs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsangnandar%2Fcsv-fixer-for-gcs/lists"}