{"id":30373416,"url":"https://github.com/dylanpicart/google_sheet_survey_pipeline","last_synced_at":"2026-05-16T01:34:34.754Z","repository":{"id":308248839,"uuid":"1032130476","full_name":"dylanpicart/google_sheet_survey_pipeline","owner":"dylanpicart","description":"Lightweight Python ETL using Google Cloud to Extract, Transform, and Load Google Surveys for Exploratory Data Analysis on Streamlit Dashboard and for Excel viewing.","archived":false,"fork":false,"pushed_at":"2025-08-06T04:24:12.000Z","size":127,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-25T21:14:00.807Z","etag":null,"topics":["api","ci","cicd","continuous-integration","data-cleaning","devsecops","etl","google","google-cloud","google-cloud-platform","python","streamlit"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dylanpicart.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-04T21:20:38.000Z","updated_at":"2025-09-18T22:04:51.000Z","dependencies_parsed_at":"2025-08-06T03:20:24.592Z","dependency_job_id":null,"html_url":"https://github.com/dylanpicart/google_sheet_survey_pipeline","commit_stats":null,"previous_names":["dylanpicart/google_sheet_survey_analysis","dylanpicart/google_sheet_survey_pipeline"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dylanpicart/google_sheet_survey_pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dylanpicart%2Fgoogle_sheet_survey_pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dylanpicart%2Fgoogle_sheet_survey_pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dylanpicart%2Fgoogle_sheet_survey_pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dylanpicart%2Fgoogle_sheet_survey_pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dylanpicart","download_url":"https://codeload.github.com/dylanpicart/google_sheet_survey_pipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dylanpicart%2Fgoogle_sheet_survey_pipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33087028,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-15T20:25:35.270Z","status":"ssl_error","status_checked_at":"2026-05-15T20:25:34.732Z","response_time":103,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","ci","cicd","continuous-integration","data-cleaning","devsecops","etl","google","google-cloud","google-cloud-platform","python","streamlit"],"created_at":"2025-08-20T08:27:10.225Z","updated_at":"2026-05-16T01:34:34.744Z","avatar_url":"https://github.com/dylanpicart.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Google Sheet Survey ETL Pipeline\n\n[![CI](https://github.com/dylanpicart/google_sheet_survey_pipeline/actions/workflows/ci.yml/badge.svg)](https://github.com/dylanpicart/google_sheet_survey_pipeline/actions/workflows/ci.yml)\n\n\nThis project provides a robust, DevSecOps-ready reproducible ETL (Extract, Transform, Load) pipeline for end-to-end analysis of multi-year, multi-language Google Sheet survey data, including raw extraction, cleaning, canonical mapping, advanced summaries, and executive-ready Excel outputs.\n\n---\n\n## **Pipeline Overview**\n\n* **Input:** Raw survey CSVs (downloaded from Google Drive or other source folders).\n* **Process:** Clean, normalize, translate, audit, map, summarize, and consolidate responses and questions.\n* **Output:** Master Excel files for analysis; fully harmonized survey data for reporting or modeling.\n\n---\n\n## Project Structure\n\n```\nroot/\n│\n├── Makefile                 # Run extract, transform, or load scripts\n├── .venv/                   # Python virtual environment \n├── creds/                   # Google API credentials\n│\n├── data/\n│   ├── configs/             # Config files (YAML, mappings, links)\n│   ├── raw/                 # All raw survey data (CSV, pre-clean)\n│   └── processed/           # All cleaned, processed, and summary data\n│\n├── notebook/                # Jupyter notebooks and experiments\n├── tests/                   # Unit tests and pipeline QA scripts\n├── utils/                   # Shared Python utilities/helpers\n│\n├── scripts/\n│   ├── extract/             # Scripts for data download, cleaning, auditing\n│   ├── transform/           # Scripts for mapping, summaries, consolidation\n│   └── load/                # Scripts for Excel/output/reporting\n│\n├── logs/                    # Logging data and issues throughout the ETL pipeline\n├── .env                     # Environment variables\n├── .gitignore               # Files and folders excluded from git\n├── README.md                # Project overview and documentation\n├── SECURITY.md              # Security and responsible disclosure policy\n└── requirements.txt         # Python dependencies\n\n```\n---\n\n## **ETL Process**\n\n**1. Extract**\n\n* Download all raw survey CSVs (English, Spanish, etc.) for all years/cohorts.\n\n### **Google API Extraction**\n\n#### **Overview**\n\nThis pipeline’s first step (`scrape_drive_links.py`) automates the downloading of raw survey CSV files from Google Drive (or optionally Google Sheets, converted to CSV).\nIt uses the Google Drive and/or Google Sheets API for authenticated access to your organization’s files and ensures you always work with up-to-date, original data.\n\n---\n\n#### **Requirements**\n\n* **Google Cloud Platform Project** with Drive and/or Sheets API enabled\n* **Service Account credentials** (or OAuth2 credentials) with access to the relevant files/folders\n* The following **Python packages**:\n\n  * `google-api-python-client`\n  * `google-auth-httplib2`\n  * `google-auth-oauthlib`\n  * `gspread` (for direct Google Sheets to CSV, optional)\n  * `pandas`\n  * `requests`\n  * `tqdm` (optional, for progress bars)\n\n**Install them with:**\n\n```bash\npip install google-api-python-client google-auth-httplib2 google-auth-oauthlib gspread pandas requests tqdm\n```\n\n---\n\n#### **Google API Setup**\n\n1. **Create a Google Cloud project** at [https://console.cloud.google.com/](https://console.cloud.google.com/)\n2. **Enable the APIs:**\n\n   * Google Drive API\n   * (Optionally) Google Sheets API\n3. **Create and download a Service Account key** (JSON) or OAuth client secret.\n4. **Share your survey folder/files** with the Service Account email (if using a Service Account).\n5. **Save your credentials** (usually as `service_account.json` or `credentials.json`) in a secure location.\n\n---\n\n#### **Script Configuration**\n\n* Place your credentials file in a known location (e.g., `secrets/service_account.json`).\n* Set an environment variable or `.env` entry:\n\n  ```\n  GOOGLE_APPLICATION_CREDENTIALS=secrets/service_account.json\n  ```\n* Update `scrape_drive_links.py` to read your folder IDs or search queries as needed.\n\n---\n\n#### **How the Extraction Script Works**\n\n* **Authenticates** with Google Drive using your credentials.\n* **Lists all files** in a given folder or matching a search query.\n* **Downloads** each file as CSV (either as a raw file or converts a Google Sheet to CSV).\n* **Saves** each file into a local directory (e.g., `raw/` or `data/raw/`).\n\n---\n\n#### **Common Usage Example**\n\n```bash\npython -m scripts.extract.scrape_drive_links\n```\n\nThis will:\n\n* Authenticate using your Google credentials\n* Download all target survey files (for each year/cohort) to your local `raw/` directory\n* Ensure that your pipeline always starts with the latest official data\n\n---\n\n#### **Possible Extraction Script Features**\n\n* **File type filtering** (only download `.csv` or `.gsheet`)\n* **Date filtering** (only new/modified files)\n* **Automatic conversion** of Google Sheets to CSV\n* **Logging of all downloaded file names, IDs, and timestamps**\n* **Parallel downloads** for large batches\n\n---\n\n#### **Error Handling**\n\n* **Missing credentials**: Script halts and prints a clear message\n* **API quota exceeded**: Script sleeps and retries (or halts with a warning)\n* **No files found**: Script exits and prints which query/folder was empty\n* **Partial download**: Script can be resumed/restarted safely\n\n---\n\n#### **Security**\n\n* **Never commit credentials** (JSON) to git or public repos\n* Restrict service account to “read-only” where possible\n\n---\n\n### **Pro Tips**\n\n* Use **Google Groups** to give bulk file access to your service account.\n* For one-off manual downloads, use Google Drive web UI, but for reproducibility, always prefer API automation.\n* Keep your service account credentials and folder IDs in `.env` or as command-line args for flexibility.\n\n---\n\n**2. Transform**\n\n* Standardize headers, clean rows, handle missing or malformed data.\n* Translate Spanish CSVs to English.\n* Audit all unique questions/options; map variations to canonical/overarching questions.\n* Generate summary tables (Likert, frequency, Yes/No, etc.) per year and group.\n* Consolidate all responses and question summaries across years and groups.\n* Summarize totals for cross-year and cross-group analysis.\n\n**3. Load**\n\n* Export all final and intermediate outputs to Excel, with tabs for every summary and year/group.\n* Provide “master summary” and “full data” workbooks for stakeholders.\n\n---\n\n## **Requirements**\n\n* **Python 3.8+**\n* **Required Python packages:**\n\n  * `pandas`\n  * `xlsxwriter`\n  * (optionally: `openpyxl`, `numpy`, `unicodedata`, `regex`)\n* **bash** (for pipeline orchestration)\n\nInstall requirements (example):\n\n```bash\npip install pandas xlsxwriter openpyxl\n```\n\n---\n\n## **How to Run the Pipeline**\n\n### **Installation and Requirements**\n\n1. **Clone this repository:**\n\n   ```bash\n   git clone \u003cyour-repo-url\u003e\n   cd \u003cyour-repo-folder\u003e\n   ```\n\n2. **Set up your Python virtual environment** (recommended):\n\n   ```bash\n   python -m venv venv\n   source venv/bin/activate  # (On Windows: venv\\Scripts\\activate)\n   ```\n\n3. **Install all Python dependencies:**\n\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n---\n\n4. **Run the pipeline with:**\n\n   ```bash\n   ./run_etl.sh\n   ```\n\n   *(Ensure it’s executable: `chmod +x run_etl.sh`)*\n\n**This will:**\n\n* Scrape and standardize raw data\n* Translate (if needed)\n* Audit, map, and consolidate all survey questions/responses\n* Output results as `SF_Master_Summary.xlsx` and `SF_Master_Data.xlsx` in the `data/processed/` directory\n\n---\n\n### **Using the Makefile for Your ETL Pipeline**\n\nThis project includes a **Makefile** for easy automation and orchestration of the ETL workflow.\n\n#### **Common Makefile Commands**\n\n* **Run the full ETL pipeline (skipping Google Drive scraping and confirming before the final Excel output):**\n\n  ```bash\n  make pipeline\n  ```\n\n* **Run only the extract step (data loading, cleaning, translation, audit):**\n\n  ```bash\n  make extract\n  ```\n\n* **Run only the transform step (mapping, summary tables, consolidation):**\n\n  ```bash\n  make transform\n  ```\n\n* **Manually run the scraping step, if you need to update the survey links:**\n\n  ```bash\n  make scrape\n  ```\n\n* **Run only the final Excel export step:**\n\n  ```bash\n  make load\n  ```\n\n* **Clean processed data outputs:**\n\n  ```bash\n  make clean\n  ```\n\n* **Run all unit tests:**\n\n  ```bash\n  make test\n  ```\n\n---\n\n#### **How it works**\n\n* The Makefile allows you to run any ETL stage individually or in sequence.\n* The default `pipeline` target skips the Google Drive scraping step and asks for confirmation before creating the final Excel output, so you won’t overwrite results by accident.\n* You can always run the original `run_etl.sh` script for fully automated execution.\n\n---\n\n#### **Pro tip:**\n\n**Always run `make` commands from the project root directory.**\nThe Makefile expects all scripts, data, and outputs to use the standard project structure.\n\n---\n\n## **Logging**\n\nAll steps of the ETL pipeline are **fully logged** for transparency, debugging, and auditability.\nLogs are stored in the `logs/` directory, with separate subfolders for each ETL phase:\n\n```\nlogs/\n├── extract/\n│   ├── extract.log           # All extraction progress and info\n│   └── extract_error.log     # Extraction warnings and errors only\n├── transform/\n│   ├── transform.log         # Data cleaning, mapping, consolidation steps\n│   └── transform_error.log   # Warnings and errors during transformation\n├── load/\n│   ├── load.log              # Final Excel/report generation steps\n│   └── load_error.log        # Any errors or warnings during loading/output\n```\n\n* **Every script in the pipeline logs to its respective ETL phase.**\n* **Info, warning, and error messages** are captured, including which files were processed, skipped, or had issues.\n* **Error logs** make it easy to spot and debug failed data loads, missing columns, or API problems.\n* Logs rotate automatically to prevent disk overflows.\n\n---\n\n## 🚦 Continuous Integration \u0026 Quality Assurance\n\nThis project uses **GitHub Actions** for full CI/CD, automated linting, and security checks:\n\n- **On every push or pull request:**\n  - All code is linted with [Ruff](https://github.com/astral-sh/ruff) and [Flake8](https://flake8.pycqa.org/).\n  - Security is checked with [Bandit](https://bandit.readthedocs.io/) and [pip-audit](https://pypi.org/project/pip-audit/).\n  - All tests are run with [pytest](https://pytest.org/), including code coverage reports.\n  - Logs and coverage reports are uploaded as build artifacts on failure.\n\nYou can view build status, logs, and artifacts on the [Actions tab](https://github.com/yourusername/google_sheet_survey_pipeline/actions).\n\n---\n\n## 🧹 Linting \u0026 Formatting\n\n**Code is autoformatted and style-enforced with:**\n- [Black](https://black.readthedocs.io/en/stable/) (for code style, line length, and blank lines)\n- [Ruff](https://github.com/astral-sh/ruff) (for code hygiene and unused code removal)\n- [Flake8](https://flake8.pycqa.org/) (for PEP8 and custom style rules)\n\n**To lint and auto-fix your code locally:**\n```bash\nblack . --line-length 120\nruff check scripts/ tests/ utils/ --fix\nflake8 scripts/ tests/ utils/\nbandit -r scripts/ utils/\n```\n---\n\n## **Possible Approaches to Data Cleaning**\n\n* **Column normalization:** Consistently rename and strip all headers (lowercase, no spaces, uniform naming).\n* **String cleaning:** Remove leading/trailing whitespace, standardize punctuation/quotes, handle Unicode artifacts.\n* **Missing value handling:** Remove or fill empty rows/cells, standardize “N/A”, handle outliers.\n* **Canonical mapping:** Use `QCON_MAP` and audit mapping to ensure all question variations are consolidated.\n* **Translation:** Ensure all responses/options are in English for cross-year consistency.\n\n---\n\n## **Next Steps (Pivot \u0026 Advanced Analysis)**\n\n* **Multi-select/multi-pivot:** Use canonical mappings to group, count, and analyze all subquestions/options as part of overarching categories.\n* **Percentages and rates:** Compute %Yes/No/Maybe per group/year/option.\n* **Longitudinal tracking:** Analyze trends in responses across years and cohorts.\n* **Visualization:** Use Excel or Python/Streamlit dashboards for charts, pivots, and interactive summaries.\n\n---\n\n## **Error Handling \u0026 Quality Assurance**\n\n* **Each ETL step is checked:** Pipeline halts if a key script or file is missing or if any step fails.\n* **Inter-step file checks:** Verifies output files from each phase before moving to the next.\n* **Logging:** All errors and successes are printed to console (or optionally, a log file).\n* **Custom checks:** Can be extended to email, Slack, or alert on failure for production.\n\n**How to debug:**\n\n* Inspect the output/error message to see where the process halted.\n* Review intermediate outputs (CSV/Excel) to verify data at each stage.\n* Re-run just the failed script for rapid iteration.\n\n---\n## License\nThis project is licensed under the MIT License. See the LICENSE file for details.\n\n---\n\n## **Author**\n\nDeveloped by **Dylan Picart at Partnership With Children.**\n\nOpen an issue or submit a pull request!\nFor help running or customizing the pipeline, contact dylanpicart@mail.adelphi.edu.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdylanpicart%2Fgoogle_sheet_survey_pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdylanpicart%2Fgoogle_sheet_survey_pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdylanpicart%2Fgoogle_sheet_survey_pipeline/lists"}