{"id":40436188,"url":"https://github.com/data-to-insight/dfe-csc-api-data-flows","last_synced_at":"2026-02-23T18:13:55.335Z","repository":{"id":310693917,"uuid":"911121932","full_name":"data-to-insight/dfe-csc-api-data-flows","owner":"data-to-insight","description":"DfE CSC Private Dashboard API - Early adopters daily data flows pilot","archived":false,"fork":false,"pushed_at":"2026-02-09T14:10:13.000Z","size":1848,"stargazers_count":0,"open_issues_count":19,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-09T18:51:25.183Z","etag":null,"topics":["api","dashboard","dfe","pilot","pipeline","powershell","python"],"latest_commit_sha":null,"homepage":"https://data-to-insight.github.io/dfe-csc-api-data-flows/","language":"TSQL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/data-to-insight.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"docs/roadmap_implementation.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-01-02T09:45:02.000Z","updated_at":"2026-02-09T14:10:17.000Z","dependencies_parsed_at":"2025-08-19T18:34:51.443Z","dependency_job_id":"3c16063b-984e-450b-a826-31b6d34a546f","html_url":"https://github.com/data-to-insight/dfe-csc-api-data-flows","commit_stats":null,"previous_names":["data-to-insight/dfe-csc-api-data-flows"],"tags_count":25,"template":false,"template_full_name":null,"purl":"pkg:github/data-to-insight/dfe-csc-api-data-flows","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/data-to-insight%2Fdfe-csc-api-data-flows","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/data-to-insight%2Fdfe-csc-api-data-flows/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/data-to-insight%2Fdfe-csc-api-data-flows/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/data-to-insight%2Fdfe-csc-api-data-flows/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/data-to-insight","download_url":"https://codeload.github.com/data-to-insight/dfe-csc-api-data-flows/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/data-to-insight%2Fdfe-csc-api-data-flows/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29750158,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-23T07:44:07.782Z","status":"ssl_error","status_checked_at":"2026-02-23T07:44:07.432Z","response_time":90,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","dashboard","dfe","pilot","pipeline","powershell","python"],"created_at":"2026-01-20T16:32:49.200Z","updated_at":"2026-02-23T18:13:55.298Z","avatar_url":"https://github.com/data-to-insight.png","language":"TSQL","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# CSC API Pipeline\n\n![Status](https://img.shields.io/badge/status-in%20development-orange)\n![Version](https://img.shields.io/badge/version-0.3.4-blue)\n![Spec](https://img.shields.io/badge/spec-0.8-blueviolet)\n![Platform](https://img.shields.io/badge/platform-Local%20Authority%20CMS-lightgrey)\n![Data%20Format](https://img.shields.io/badge/data%20format-JSON%20%7C%20SQL-yellow)\n![DfE%20API](https://img.shields.io/badge/DfE%20API-Connected-brightgreen)\n![SSD%20Compatible](https://img.shields.io/badge/SSD-Compatible-success)\n![Automation](https://img.shields.io/badge/Automation-Enabled-informational)\n\nA Python and equivellent Powershell(non-delta payloads only) tools for submitting agreed structured JSON payloads to the DfE Children’s Social Care (CSC) API endpoint. Expected to run as scheduled job (as part of LAs existing overnight batches). However release files also offer scope for LA deployment|project teams to fully end-to-end test locally, with only minimal privilleges. \n\n---\n\n## Features\n\n- Extracts data from a SQL Server SSD staging table\n- Generates full or partial JSON payloads based on change detection\n- Authenticates via OAuth2 client credentials flow\n- Submits payloads to an API endpoint with retry and backoff logic\n- Logs responses and updates submission state in SQL\n- Offers `.exe` packaging for offline deployment via PyInstaller\n- Includes CLI tools for test and verification modes\n- Project published help and how-to technical documentation\n\n---\n\n## Supported platforms and databases\n\n- Python 3.10, 3.11, 3.12\n- Windows and Linux, ODBC required\n- Microsoft SQL Server 2012, and 2016 SP1 compatible\n- Aligns with the Standard Safeguarding Dataset, SSD schema\n\n\n## Installation\n\nClone the repo and install locally:\n\n```bash\npip install .\n```\n\nOr run directly as a module:\n\n```bash\npython -m api_pipeline run\n```\n\n---\n\n## Environment Config\n\nCopy or rename `.env.example` and fill in missing values using your DfE supplied values:\n\n```ini\nDB_CONNECTION_STRING=...\nTOKEN_ENDPOINT=...\nAPI_ENDPOINT=...\nCLIENT_ID=...\nCLIENT_SECRET=...\nSCOPE=...\nSUPPLIER_KEY=...\nDEBUG=true\n```\n\n---\n## Pre flight checks\n\nA separate archive, `pre_flight_checks.zip`, contains:\n- `phase_1_api_credentials_smoke_test.ps1`\n- `ssd_csc_api_schema_checks.sql`\n\nDownload, extract, run the smoke test first, then run schema checks against your deployed SSD schema.\n\n\n## Run API Pipeline\n\n### As a Python module\n\n```bash\npython -m api_pipeline run\n```\n\n### As executable (packaged `.exe`):\n\n```bash\ncsc_api_pipeline.exe\n```\n\n\u003e **Note for Windows users:**  \n\u003e The executable bundles Python and all required libraries, but it still requires the **Microsoft ODBC Driver for SQL Server** to be installed on your machine. We expect most LA colleagues will have this by default. But if not:   \n\u003e Download and install **ODBC Driver 18 for SQL Server** from the official Microsoft site:  \n\u003e \u003chttps://learn.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server\u003e  \n\u003e Without this driver, the `.exe` won't be able to connect to your database.\n\u003e\n\u003e **Configuration:** Place your `.env` file (based on `.env.example`) in the same folder where you run the `.exe`. The tool will read from current working directory.\n\n\n---\n\n## CLI Commands\n\nRun local diagnostics (limited options at the moment):\n\n```bash\npython -m api_pipeline test-endpoint\npython -m api_pipeline test-db-connection\npython -m api_pipeline test-schema\n```\n\nUsing the installed CLI (after pip install):\n\n```bash\nd2i-csc-api run\nd2i-csc-api test-endpoint\nd2i-csc-api test-db-connection\nd2i-csc-api test-schema\n```\n\nFrom the Windows executable:\n\n```bash\ncsc_api_pipeline.exe test-endpoint\ncsc_api_pipeline.exe test-db-connection\ncsc_api_pipeline.exe test-schema\n```\n\n## CLI Help\n\n```bash\nd2i-csc-api --help\ncsc_api_pipeline.exe --help\n```\n\nReturns:\n\n```\nCSC API Pipeline CLI\n\nUsage:\n  d2i-csc-api [command]\n  csc_api_pipeline.exe [command]\n\nCommands:\n  run             Run full API submission process\n  test-endpoint   Check API connectivity and token authentication\n  test-db-connection   Check database connection\n  test-schema     Validate required table structure/schema\n```\n\nOn Windows user installs, ensure your Python scripts directory is on PATH (e.g. %APPDATA%\\Python\\Python310\\Scripts), or call inside a virtual env.\n\n---\n\n## Verifying Executable Contents\n\nTo help confirm that downloaded `.exe` is safe and built from the provided Python source code:\n\n### Check the SHA256 Hash\n\nEach file on [GitHub Releases page](https://github.com/data-to-insight/dfe-csc-api-data-flows/releases) includes a `sha256:` hash. After downloading, you can verify file integrity with:\n\n#### On Windows PowerShell:\n\n```powershell\nGet-FileHash .\\csc_api_pipeline.exe -Algorithm SHA256\n```\n\n#### On Linux/macOS:\n\n```bash\nshasum -a 256 csc_api_pipeline.exe\n```\n\nCompare this output to hash shown next to `.exe` in the release.\n\n### Rebuild Locally (Advanced)\n\nIf needed, you can rebuild the `.exe` yourself from source:\n\n```bash\ngit clone https://github.com/data-to-insight/dfe-csc-api-data-flows.git\ncd dfe_csc_api_data_flows\npip install -r requirements.txt\npyinstaller api_pipeline/entry_point.py --onefile --name csc_api_pipeline\n```\n\nThen compare hash of the new `dist/csc_api_pipeline.exe` with downloaded one.\n\nThis ensures full transparency of the release process.\n\n---\n\n## (D2I Dev only)Documentation\n\nDocumentation is maintained under `docs/` and built via MkDocs\n\n- **Local preview:** `mkdocs serve`\n- **Static build:** `mkdocs build`\n- **PDF export:** `mkdocs-with-pdf build`\n\nSite (and release updates) auto-deploys via Git Pages on push to `main` into : [data-to-insight.github.io/dfe-csc-api-data-flows](https://data-to-insight.github.io/dfe-csc-api-data-flows/)\n\nWorkflows:\n- `.github/workflows/build-api-pipeline-pkg.yml` – build Py distribution artifacts (sdist \u0026 wheel) and uploads as Git Actions artifact\n- `.github/workflows/build-release-exe.yml` – build Windows exe\n- `.github/workflows/gh-pages.yml` – build and deploy MkDocs site\n\nNote: `pyproject.toml` and `MANIFEST.in` live at project root (for PEP621 + package-data inclusion).  \n\n\n---\n\n## Git Actions Workflows\n\n1. **build-api-pipeline-pkg.yml** – Build `.whl` and `.tar.gz` Python packages and upload as Git Actions artifacts (triggered by push or PR to `main`)\n2. **build-release-exe.yml** – On version tag push or Git Release, build Windows `.exe`, package into `release.zip`, include `.env.example`, README.md, plus `.whl` and `.tar.gz`, then upload to Git Release\n3. **gh-pages.yml** – On push to `main`, build and deploy MkDocs site to Git Pages\n\n\n## Git Actions Diagram\n\n```mermaid\n\nflowchart LR\n  %% Triggers\n  A1[Push to main]\n  A2[Pull request to main]\n  B1[Version tag push v1.2.3]\n  B2[GitHub Release event]\n\n  %% Workflows\n  W1[build-api-pipeline-pkg.yml]\n  W2[build-release-exe.yml]\n  W3[gh-pages.yml]\n\n  %% Outputs\n  O11[Wheel .whl]\n  O12[Source tarball .tar.gz]\n\n  O21[csc_api_pipeline.exe]\n  O22[release.zip]\n  O23[Wheel and sdist]\n  O24[.env.example]\n  O25[README.md]\n\n  O31[Deployed MkDocs site]\n\n  %% Edges\n  A1 --\u003e W1\n  A2 --\u003e W1\n  B1 --\u003e W2\n  B2 --\u003e W2\n  A1 --\u003e W3\n\n  W1 --\u003e O11\n  W1 --\u003e O12\n\n  W2 --\u003e O21\n  W2 --\u003e O22\n  W2 --\u003e O23\n  W2 --\u003e O24\n  W2 --\u003e O25\n\n  W3 --\u003e O31\n\n```\n\n---\n\n## Project Structure\n\n| File / Folder           | Purpose                               |\n|-------------------------|----------------------------------------|\n| `api_pipeline/`         | Core pipeline code                     |\n| `api_pipeline/entry_point.py` | Standalone launcher             |\n| `api_pipeline/config.py` | Loads environment and config values    |\n| `api_pipeline/auth.py`   | Handles OAuth authentication           |\n| `api_pipeline/db.py`     | Manages SQL Server access              |\n| `api_pipeline/payload.py`| Builds JSON payloads                   |\n| `api_pipeline/tests.py`  | Diagnostics and CLI tools              |\n| `docs/`                 | MkDocs site content                    |\n| `.github/workflows/`    | CI and deployment definitions          |\n| `pyproject.toml`        | Packaging metadata and build-system    |\n| `MANIFEST.in`           | Include non-code files in sdist        |\n| `README.md`             | This file                              |\n| `LICENSE`               | Licensing information                  |\n\n\n---\n\n## License\n\nMIT License. See `LICENSE` file.\n\n---\n\n## (D2I dev) Notes\n\n- Run using `python -m api_pipeline` in dev\n- Maintain `.env` in root for local config\n- Build Windows executable using:\n\n```bash\npyinstaller api_pipeline/entry_point.py --onefile --name csc_api_pipeline\n```\n\n```bash\npython -m build --sdist --wheel --outdir dist\n```\n\n**- Check environment before release:**\n\n```bash\ngitleaks detect --no-git --source=. --report-path=current_only.json\n```\n\n\n**- Release Process (D2I dev)**\n\nUse `./release.sh` to bump version, build Python package (sdist + wheel), (optionally) build `.exe`, zip artifacts, tag, and push.  \nTrigger workflows via pushing a `vX.Y.Z` tag or publishing a Release in GitHub UI.\n\n---\n\n## Badges\n\n![Python](https://img.shields.io/badge/python-3.10%E2%80%933.12-blue)\n![Build](https://github.com/data-to-insight/dfe_csc_api_data_flows/actions/workflows/gh-pages.yml/badge.svg)\n![License](https://img.shields.io/github/license/data-to-insight/dfe-csc-api-data-flows)\n\n---\n\n## Support\n\nFor technical issues, raise a GitHub Issue(not yet enabled) or contact the D2I API team.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdata-to-insight%2Fdfe-csc-api-data-flows","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdata-to-insight%2Fdfe-csc-api-data-flows","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdata-to-insight%2Fdfe-csc-api-data-flows/lists"}