{"id":33585399,"url":"https://github.com/windy-civi-pipelines/or-data-pipeline","last_synced_at":"2026-06-14T08:02:39.785Z","repository":{"id":320510530,"uuid":"1082330511","full_name":"windy-civi-pipelines/or-data-pipeline","owner":"windy-civi-pipelines","description":"🏛️ Oregon legislative data pipeline","archived":false,"fork":false,"pushed_at":"2026-05-31T06:26:15.000Z","size":248,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-31T08:19:06.554Z","etag":null,"topics":["legislative-data","openstates","state-pipeline"],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/windy-civi-pipelines.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-24T04:47:43.000Z","updated_at":"2025-10-30T18:39:44.000Z","dependencies_parsed_at":"2026-01-09T10:06:35.861Z","dependency_job_id":null,"html_url":"https://github.com/windy-civi-pipelines/or-data-pipeline","commit_stats":null,"previous_names":["windy-civi-pipelines/or-data-pipeline"],"tags_count":217,"template":false,"template_full_name":"windy-civi-pipelines/windy-civi-template-pipeline","purl":"pkg:github/windy-civi-pipelines/or-data-pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/windy-civi-pipelines%2For-data-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/windy-civi-pipelines%2For-data-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/windy-civi-pipelines%2For-data-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/windy-civi-pipelines%2For-data-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/windy-civi-pipelines","download_url":"https://codeload.github.com/windy-civi-pipelines/or-data-pipeline/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/windy-civi-pipelines%2For-data-pipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34234557,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-12T02:00:06.859Z","response_time":109,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["legislative-data","openstates","state-pipeline"],"created_at":"2025-11-29T07:03:41.878Z","updated_at":"2026-06-12T08:01:32.804Z","avatar_url":"https://github.com/windy-civi-pipelines.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🏛️ Windy Civi Data Pipeline Template\n\nA **GitHub Actions-powered pipeline** that scrapes, cleans, versions, and extracts text from state legislative data from **Open States**. This repository acts as a standardized template for all state-level pipelines within the Windy Civi ecosystem.\n\n---\n\n## ⚙️ What This Pipeline Does\n\nEach state pipeline provides a self-contained automation workflow to:\n\n1. 🧹 **Scrape** data for a single U.S. state from the [OpenStates](https://github.com/openstates/openstates-scrapers) project\n2. 🧼 **Sanitize** the data by removing ephemeral fields (`_id`, `scraped_at`) for deterministic output\n3. 🧠 **Format** it into a blockchain-style, versioned structure with incremental processing\n4. 🔗 **Link** events to bills and sessions automatically\n5. 🩺 **Monitor** data quality by tracking orphaned bills\n6. 📄 **Extract** full text from bills, amendments, and supporting documents (PDFs, XMLs, HTMLs)\n7. 📂 **Commit** the formatted output and extracted text nightly (or manually) with auto-save\n\nThis approach keeps every state repository consistent, auditable, and easy to maintain.\n\n---\n\n## ✨ Key Features\n\n- **🔄 Incremental Processing** - Only processes new or updated bills (no duplicate work!)\n- **💾 Auto-Save Failsafe** - Commits progress every 30 minutes during text extraction\n- **🩺 Data Quality Monitoring** - Tracks orphaned bills (votes/events without bill data)\n- **🔗 Bill-Event Linking** - Automatically connects committee hearings and events to bills\n- **⏱️ Timestamp Tracking** - Two-level timestamps for logs and text extraction\n- **🎯 Multi-Format Text Extraction** - XML → HTML → PDF with fallbacks\n- **🔀 Concurrent Job Support** - Multiple runs can safely update the same repository\n- **📊 Detailed Error Logging** - Categorized errors for easy debugging\n\n---\n\n## 🔧 Setup Instructions\n\n1. **Click the green \"Use this template\" button** on this repository page to create a new repository from this template.\n\n2. **Name your new repository** using the convention: `Oregon Data Pipeline` (e.g., `il-data-pipeline`, `tx-data-pipeline`).\n\n3. **Update the state abbreviation** in both workflow files:\n\n   **In `.github/workflows/scrape-and-format-data.yml`:**\n\n   ```yaml\n   env:\n     STATE_CODE: or # CHANGE THIS to your state abbreviation\n\n   jobs:\n     scrape:\n       - name: Scrape data\n         uses: windy-civi/toolkit/actions/scrape@main\n         with:\n           state: ${{ env.STATE_CODE }}\n\n     format:\n       - name: Format data\n         uses: windy-civi/toolkit/actions/format@main\n         with:\n           state: ${{ env.STATE_CODE }}\n   ```\n\n   **In `.github/workflows/extract-text.yml`:**\n\n   ```yaml\n   - name: Extract text\n     uses: windy-civi/toolkit/actions/extract@main\n     with:\n       state: or # CHANGE THIS to your state abbreviation\n   ```\n\n   Make sure the state abbreviation matches the folder name used in [Open States scrapers](https://github.com/openstates/openstates-scrapers/tree/main/scrapers).\n\n4. **Enable GitHub Actions** in your repo (if not already enabled).\n\n5. (Optional) Enable nightly runs by ensuring the schedule blocks are uncommented in both workflow files:\n\n   ```yaml\n   on:\n     workflow_dispatch:\n     schedule:\n       - cron: \"0 1 * * *\" # For scrape-and-format-data.yml\n       # or\n       - cron: \"0 3 * * *\" # For extract-text.yml (runs later to avoid overlap)\n   ```\n\n---\n\n## 📅 Workflow Schedule\n\nThe pipeline runs in two stages:\n\n### **Stage 1: Scrape \u0026 Format** (1am UTC)\n\nTwo separate jobs that run sequentially:\n\n1. **Scrape Job** - Downloads legislative data using OpenStates scrapers\n2. **Format Job** - Processes scraped data, links events, and monitors quality\n\n### **Stage 2: Text Extraction** (3am UTC)\n\nIndependent workflow that extracts full bill text from documents.\n\nThis separation allows:\n\n- ✅ Faster metadata updates\n- ✅ Independent monitoring and debugging\n- ✅ Text extraction can timeout and restart without affecting scraping\n- ✅ Better resource management (text extraction can take hours)\n\n---\n\n## 📁 Folder Structure\n\n```\nOregon Data Pipeline/\n├── .github/workflows/\n│   ├── scrape-and-format-data.yml  # Metadata scraping + formatting\n│   └── extract-text.yml             # Text extraction (independent)\n├── country:us/\n│   └── state:xx/                    # state:usa for federal, state:il for Illinois, etc.\n│       └── sessions/\n│           └── {session_id}/\n│               ├── bills/\n│               │   └── {bill_id}/\n│               │       ├── metadata.json      # Bill data + _processing timestamps\n│               │       ├── files/             # Extracted text \u0026 documents\n│               │       │   ├── *.pdf          # Original PDFs\n│               │       │   ├── *.xml          # Original XMLs\n│               │       │   └── *_extracted.txt # Extracted text\n│               │       └── logs/              # Action/event/vote logs\n│               └── events/                    # Committee hearings\n│                   └── {timestamp}_hearing.json\n├── .windycivi/                      # Pipeline metadata (committed)\n│   ├── errors/                      # Processing errors\n│   │   ├── text_extraction_errors/  # Text extraction failures\n│   │   │   ├── download_failures/   # Failed downloads\n│   │   │   ├── parsing_errors/      # Failed text parsing\n│   │   │   └── missing_files/       # Missing source files\n│   │   ├── missing_session/         # Bills without session info\n│   │   ├── event_archive/           # Archived event data\n│   │   └── orphaned_placeholders_tracking.json  # Data quality monitoring\n│   ├── bill_session_mapping.json    # Bill-to-session mappings (flattened)\n│   ├── sessions.json                # Session metadata (flattened)\n│   └── latest_timestamp_seen.txt    # Last processed timestamp\n├── Pipfile, Pipfile.lock\n└── README.md\n```\n\n---\n\n## 📦 Output Format\n\n### Metadata Output (`country:us/state:*/`)\n\nFormatted metadata is saved to `country:us/state:xx/sessions/`, organized by session and bill.\n\nEach bill directory contains:\n\n- `metadata.json` – structured information about the bill **with `_processing` timestamps**\n- `logs/` – action, event, and vote logs\n- `files/` – original documents and extracted text\n\n**Example `metadata.json` structure:**\n\n```json\n{\n  \"identifier\": \"HB 1234\",\n  \"title\": \"Example Bill\",\n  \"_processing\": {\n    \"logs_latest_update\": \"2025-01-15T14:30:00Z\",\n    \"text_extraction_latest_update\": \"2025-01-16T08:00:00Z\"\n  },\n  \"actions\": [\n    {\n      \"description\": \"Introduced in House\",\n      \"date\": \"2025-01-01\",\n      \"_processing\": {\n        \"log_file_created\": \"2025-01-01T12:00:00Z\"\n      }\n    }\n  ]\n}\n```\n\n### Text Extraction Output (`files/`)\n\nWhen text extraction is enabled, each bill directory also includes:\n\n- `files/` – original documents and extracted text\n  - `*.pdf` – Original PDF documents\n  - `*.xml` – Original XML bill text\n  - `*.html` – Original HTML documents\n  - `*_extracted.txt` – Plain text extracted from documents\n\n### Error Output (`.windycivi/errors/`)\n\nFailed items are logged separately:\n\n- `.windycivi/errors/text_extraction_errors/download_failures/` – Documents that couldn't be downloaded\n- `.windycivi/errors/text_extraction_errors/parsing_errors/` – Documents that couldn't be parsed\n- `.windycivi/errors/text_extraction_errors/missing_files/` – Bills missing source files\n- `.windycivi/errors/missing_session/` – Bills without session information\n\n### Data Quality Monitoring (`orphaned_placeholders_tracking.json`)\n\nThe pipeline automatically tracks **orphaned bills** - bills that have vote events or hearings but no actual bill data. Check this file periodically to identify data quality issues:\n\n```json\n{\n  \"HB999\": {\n    \"first_seen\": \"2025-01-21T12:00:00Z\",\n    \"last_seen\": \"2025-01-23T14:30:00Z\",\n    \"occurrence_count\": 3,\n    \"session\": \"103\",\n    \"vote_count\": 2,\n    \"event_count\": 0,\n    \"path\": \"country:us/state:il/sessions/103/bills/HB999\"\n  }\n}\n```\n\n**What to look for:**\n\n- Bills with high `occurrence_count` (3+) are **chronic orphans** - likely data quality issues\n- Check for typos in bill identifiers or scraper configuration\n- Orphans automatically resolve when the bill data arrives! 🎉\n\n📖 See [orphan tracking documentation](https://github.com/windy-civi/toolkit/blob/main/docs/orphan_tracking.md) for more details.\n\n---\n\n## 🪵 Logging \u0026 Error Handling\n\nEach run includes detailed logs to track progress and capture failures:\n\n### Scraping \u0026 Formatting Logs\n\n- Logs are saved per bill under `logs/`\n- Processing summary shows total bills, events, and votes processed\n- Session mapping tracks bill-to-session relationships\n- **Orphan tracking** shows new, existing, and resolved orphans\n\n### Text Extraction Logs\n\n- Download attempts with success/failure status\n- Extraction method used (XML, HTML, PDF)\n- Error details saved to `text_extraction_errors/`\n- **Auto-save commits** every 30 minutes prevent data loss\n- Summary reports include:\n  - Total documents processed\n  - Successful extractions by type\n  - Skipped (already extracted) documents\n  - Failed downloads/extractions with reasons\n\nPipelines are fault-tolerant — if a bill fails, the workflow continues for all others.\n\n---\n\n## 📄 Supported Document Types\n\nThe text extraction workflow supports:\n\n| Type           | Format   | Extraction Method   | Notes                          |\n| -------------- | -------- | ------------------- | ------------------------------ |\n| **Bills**      | XML      | Direct XML parsing  | Primary bill text              |\n| **Bills**      | PDF      | pdfplumber + PyPDF2 | With strikethrough detection   |\n| **Bills**      | HTML     | BeautifulSoup       | Fallback for HTML-only sources |\n| **Amendments** | PDF      | pdfplumber + PyPDF2 | State amendments only          |\n| **Documents**  | PDF/HTML | Auto-detect         | CBO reports, committee reports |\n\n**Note**: Federal `congress.gov` HTML amendments are currently skipped due to blocking issues. XML bill versions from `govinfo.gov` work perfectly.\n\n---\n\n## 🔧 Workflow Configuration Options\n\n### Scrape Action Inputs\n\n```yaml\nuses: windy-civi/toolkit/actions/scrape@main\nwith:\n  state: or # State abbreviation (required)\n  github-token: ${{ secrets.GITHUB_TOKEN }}\n  use-scrape-cache: \"false\" # Skip scraping, use cached data\n```\n\n### Format Action Inputs\n\n```yaml\nuses: windy-civi/toolkit/actions/format@main\nwith:\n  state: or # State abbreviation (required)\n  github-token: ${{ secrets.GITHUB_TOKEN }}\n```\n\n### Text Extraction Action Inputs\n\n```yaml\nuses: windy-civi/toolkit/actions/extract@main\nwith:\n  state: or # State abbreviation (required)\n  github-token: ${{ secrets.GITHUB_TOKEN }}\n```\n\n---\n\n## 🧩 Optional: Enabling Raw Scraped Data Storage\n\nBy default, raw scraped data (`_data/`) is not stored to keep the repository lightweight.\n\n### ✅ To Enable `_data` Saving:\n\nUncomment the copy and commit steps in your workflow file:\n\n```yaml\n- name: Copy Scraped Data to Repo\n  run: |\n    mkdir -p \"$GITHUB_WORKSPACE/_data/$STATE\"\n    cp -r \"${RUNNER_TEMP}/_working/_data/$STATE\"/* \"$GITHUB_WORKSPACE/_data/$STATE/\"\n```\n\nAnd include `_data` in the commit:\n\n```bash\ngit add _data country:us/ .windycivi/\n```\n\n### 🚫 To Disable `_data` Saving (Default):\n\nComment out the copy step and exclude `_data` from the commit command:\n\n```bash\ngit add country:us/ .windycivi/\n```\n\n---\n\n## 🚀 Running the Pipeline\n\n### Automatic (Scheduled)\n\nOnce enabled, workflows run automatically:\n\n- **Scrape \u0026 Format**: 1am UTC daily\n- **Text Extraction**: 3am UTC daily (runs independently)\n\n### Manual Trigger\n\n1. Go to **Actions** tab in GitHub\n2. Select the workflow (Scrape \u0026 Format or Extract Text)\n3. Click **Run workflow**\n4. Choose the branch and click **Run**\n\n### Testing Locally\n\n```bash\n# Clone the repository\ngit clone https://github.com/YOUR-ORG/Oregon Data Pipeline\ncd Oregon Data Pipeline\n\n# Install dependencies\npipenv install\n\n# Run scraping and formatting\npipenv run python scrape_and_format/main.py \\\n  --state il \\\n  --openstates-data-folder /path/to/scraped/data \\\n  --git-repo-folder /path/to/output\n\n# Run text extraction (with incremental flag)\npipenv run python text_extraction/main.py \\\n  --state il \\\n  --data-folder /path/to/output \\\n  --output-folder /path/to/output \\\n  --incremental\n```\n\n---\n\n## 🔍 Known Issues\n\nSee the [known_problems/](https://github.com/windy-civi/toolkit/tree/main/known_problems) directory in the main repository for:\n\n- State-specific scraper issues\n- Formatter validation issues\n- Text extraction limitations\n- Status of all 56 jurisdictions\n\n---\n\n## 📊 Monitoring \u0026 Debugging\n\n### Check Workflow Status\n\n- GitHub Actions tab shows all runs\n- Green checkmark = success\n- Red X = failure (click for logs)\n\n### Check Data Quality\n\n1. Review `.windycivi/errors/orphaned_placeholders_tracking.json` for data issues\n2. Look for chronic orphans (occurrence_count \u003e= 3)\n3. Check `.windycivi/errors/` for formatting/extraction errors\n4. Monitor auto-save commits during text extraction runs\n\n### Common Issues\n\n**Scraping fails**:\n\n- Check if OpenStates scraper for your state is working\n- Verify state abbreviation matches OpenStates format\n- Check for new legislative sessions not yet configured\n\n**Text extraction fails or times out**:\n\n- Check `.windycivi/errors/text_extraction_errors/` for details\n- Look for auto-save commits (pipeline saves progress every 30 minutes)\n- Re-run the workflow - it will resume from where it left off (incremental)\n- Review error logs for specific bills\n\n**Orphaned bills appear**:\n\n- Check `orphaned_placeholders_tracking.json` for details\n- Verify bill identifiers match between scraper and vote/event data\n- Bills may auto-resolve on next scrape if it's a timing issue\n\n**Push conflicts**:\n\n- The pipeline auto-handles conflicts with `git pull --rebase`\n- If manual resolution needed, check logs for specific conflicts\n\n---\n\n## 🤝 Contributions \u0026 Support\n\nThis template is part of the [Windy Civi](https://github.com/windy-civi) project. If you're onboarding a new state or improving the automation, feel free to open an issue or PR.\n\n**Main Repository**: https://github.com/windy-civi/toolkit\n\nFor discussions, join our community on Slack or GitHub Discussions.\n\n---\n\n## 🎯 Next Steps After Setup\n\n1. ✅ Verify both workflows are enabled\n2. ✅ Test with manual trigger first (start with Scrape \u0026 Format)\n3. ✅ Check output in `country:us/state:xx/sessions/`\n4. ✅ Review `.windycivi/errors/orphaned_placeholders_tracking.json` for data quality\n5. ✅ Check any errors in `.windycivi/errors/`\n6. ✅ Test text extraction workflow independently\n7. ✅ Enable scheduled runs once testing is successful\n8. ✅ Monitor first few automated runs for issues\n\n---\n\n## 📚 Additional Documentation\n\n- **[Incremental Processing Guide](https://github.com/windy-civi/toolkit/blob/main/docs/incremental_processing/)** - How incremental updates work\n- **[Orphan Tracking Guide](https://github.com/windy-civi/toolkit/blob/main/docs/orphan_tracking.md)** - Understanding data quality monitoring\n- **[Main Repository README](https://github.com/windy-civi/toolkit)** - Full technical documentation\n\n---\n\n**Part of the [Windy Civi](https://windycivi.com) ecosystem — building a transparent, verifiable civic data archive for all 50 states.**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwindy-civi-pipelines%2For-data-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwindy-civi-pipelines%2For-data-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwindy-civi-pipelines%2For-data-pipeline/lists"}