{"id":49821528,"url":"https://github.com/coeusyk/opencast","last_synced_at":"2026-05-13T11:35:15.992Z","repository":{"id":355220208,"uuid":"1223206010","full_name":"coeusyk/opencast","owner":"coeusyk","description":"Chess opening analytics pipeline — monthly win-rate forecasting, engine-human delta scoring, and AI-generated insights across 500+ ECO openings from Lichess data.","archived":false,"fork":false,"pushed_at":"2026-05-09T19:34:37.000Z","size":12054,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-09T20:38:45.451Z","etag":null,"topics":["arima","chess","chess-analytics","data-pipeline","lichess-api","python","rust","stockfish","time-series"],"latest_commit_sha":null,"homepage":"https://coeusyk.github.io/opencast/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/coeusyk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-28T05:25:09.000Z","updated_at":"2026-05-09T08:15:31.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/coeusyk/opencast","commit_stats":null,"previous_names":["coeusyk/opencast"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/coeusyk/opencast","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coeusyk%2Fopencast","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coeusyk%2Fopencast/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coeusyk%2Fopencast/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coeusyk%2Fopencast/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/coeusyk","download_url":"https://codeload.github.com/coeusyk/opencast/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coeusyk%2Fopencast/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32980864,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-13T11:31:52.688Z","status":"ssl_error","status_checked_at":"2026-05-13T11:31:52.072Z","response_time":115,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arima","chess","chess-analytics","data-pipeline","lichess-api","python","rust","stockfish","time-series"],"created_at":"2026-05-13T11:35:04.372Z","updated_at":"2026-05-13T11:35:12.026Z","avatar_url":"https://github.com/coeusyk.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OpenCast — Chess Opening Analytics\n\nOpenCast is a data pipeline that fetches monthly win-rate snapshots from the Lichess Opening Explorer API, builds per-opening time series, forecasts future win rates, and computes an engine-human delta score - the gap between Stockfish expectation and observed human results at 2000-rated blitz. Unlike a simple leaderboard, OpenCast highlights where humans systematically diverge from engine expectation and whether those gaps are widening or narrowing.\n\n![GitHub Pages](https://img.shields.io/github/deployments/coeusyk/opencast/github-pages?label=pages\u0026style=flat-square)\n![Last Update](https://img.shields.io/github/last-commit/coeusyk/opencast/main?label=last+update\u0026style=flat-square)\n![Python](https://img.shields.io/badge/python-3.11%2B-blue?style=flat-square\u0026logo=python\u0026logoColor=white)\n![Rust](https://img.shields.io/badge/rust-1.75%2B-orange?style=flat-square\u0026logo=rust\u0026logoColor=white)\n![License](https://img.shields.io/github/license/coeusyk/opencast?style=flat-square)\n\n---\n\n## Live Dashboard\n\n[![Live Dashboard](https://img.shields.io/badge/dashboard-live-brightgreen?style=flat-square\u0026logo=github)](https://coeusyk.github.io/opencast/)\n\nDashboard is published via GitHub Pages on each pipeline run.\n\n---\n\n## Latest Findings\n\nSee [findings/findings.md](findings/findings.md) and [findings/findings.json](findings/findings.json) - auto-generated by the pipeline.\n\n## Recent Product Updates\n\n- Opening detail pages now include an **Analysis** section, trend-driver table polish, and an upgraded **Engine vs Human** card layout.\n- Trend-line rendering is restored across opening charts, with confidence-based opacity.\n- Structural-break vertical markers were removed from per-opening charts to reduce clutter.\n- Track 3 foundation is now in place:\n  - Curated opening lines live in `data/opening_lines.json`.\n  - `run_visualizer()` copies lines to `data/output/dashboard/assets/opening_lines.json`.\n  - Opening pages can render an interactive board with start/back/next controls when a curated line is available.\n  - Board coordinate labels render outside the board frame (not over piece squares).\n- Track 5 hardening is in place for the v1.0.0 push:\n  - Schema contracts live in `SCHEMAS.md`.\n  - The release guide is in `READING_OPENCAST.md`.\n  - CI now enforces a configurable runtime budget from `config.json`.\n\n\n## How It Works\n\n1. **Catalog \u0026 Selection** - `scripts/build_catalog.py` maintains the full ECO catalog (`data/openings_catalog.csv`). `src/select_openings.py` and `scripts/compute_selection_flags.py` classify openings into Tier 1/2/3 from coverage and activity thresholds.\n2. **Fetch (Rust)** - `fetcher` queries `explorer.lichess.ovh` month-by-month and stores one consolidated JSON per ECO at `data/raw/{group}/{ECO}.json` (e.g. `data/raw/A/A00.json`) with `months` and `_meta.skipped_months`.\n3. **Bootstrap Expansion** - `scripts/temp_bootstrap_openings.py` activates selected ECO batches, fetches missing months ECO-by-ECO, applies early-stop/coverage pruning, and persists fetch completion tracking (`bootstrap_fetch_complete`, `bootstrap_fetched_until`, `bootstrap_fetch_status`) in the catalog.\n4. **Freshness Guard (main.py)** - `main.py` detects missing complete months from `config.json::fetch_start` through the latest complete month, and can auto-fetch when `AUTO_FETCH_MISSING_DATA=true`. In non-interactive runs (CI), fetch is disabled unless explicitly enabled.\n5. **Ingest** - `src/ingest.py` normalizes consolidated raw files into `data/processed/openings_ts.csv`.\n6. **Analyze** - `src/timeseries.py` fits ARIMA (Tier 1) and Holt-Winters (Tier 2), then writes forecasts to `data/output/forecasts.csv`. `src/engine_delta.py` computes engine-human deltas in `data/output/engine_delta.csv` and skips malformed SAN move tokens instead of aborting the whole stage.\n7. **Report \u0026 Visualize** - `src/report.py` writes `findings/findings.md` and `findings/findings.json` (Gemini-assisted with template fallback). `src/visualizer.py` generates the static dashboard site in `data/output/dashboard/`, including tier tags on opening detail pages.\n\n---\n\n## Setup\n\n```bash\ngit clone https://github.com/coeusyk/opencast.git\ncd opencast\n\n# Install Cargo/Rust toolchain if not already installed\ncommand -v cargo \u003e/dev/null 2\u003e\u00261 || sudo apt install -y cargo rustc\n\n# Create local environment file from template (if needed)\ncp -n .env.example .env\n\n# Lichess API token (free at https://lichess.org/account/oauth/token)\nexport LICHESS_TOKEN=\u003cyour_token\u003e\n\n# Gemini API key (optional, for Gemini-generated findings)\nexport GEMINI_API_KEY=\u003cyour_gemini_api_key\u003e\n\n# Groq API key (optional, for fast LLM inference)\nexport GROQ_API_KEY=\u003cyour_groq_api_key\u003e\n\n# Build the Rust fetcher\ncd fetcher \u0026\u0026 cargo build --release \u0026\u0026 cd ..\n\n# Create and activate Python virtual environment with uv\nuv venv .venv\nsource .venv/bin/activate\n\n# Install Python dependencies with uv\nuv pip install -r requirements.txt\n\n# Run the full pipeline\npython main.py\n\n# Optional: force non-interactive mode to skip auto-fetch prompts\nAUTO_FETCH_MISSING_DATA=false python main.py\n\n# Optional: run remaining bootstrap openings after an initial batch\npython scripts/temp_bootstrap_openings.py --apply --eco-offset 240\n```\n\n\u003e **Stockfish 16** must be installed separately: `sudo apt install stockfish`\n\n\u003e **Gemini API key** (optional): `GEMINI_API_KEY` in `.env` powers AI-generated findings. `report.py` falls back to templated text if the key is absent.\n\n\u003e **Groq API key** (optional): `GROQ_API_KEY` in `.env` enables fast LLM inference as an alternative backend.\n\n---\n\n## Data Coverage\n\n![ECO Codes](https://img.shields.io/badge/ECO_codes-498-informational?style=flat-square)\n![Tiers](https://img.shields.io/badge/tiers-1%20%7C%202%20%7C%203-blueviolet?style=flat-square)\n![Date Range](https://img.shields.io/badge/data-2023--01_→_present-lightgrey?style=flat-square)\n\n| Metric | Value |\n|---|---|\n| Catalog size | 498 ECO codes |\n| Tracking scope | ECO A-E, tiered by activity/coverage |\n| Date range | 2023-01 → present |\n| Raw JSON files | one consolidated file per ECO in `data/raw/{A-E}/{ECO}.json` |\n| Processed rows | one row per ECO-month in `data/processed/openings_ts.csv` |\n| Forecast horizon | 3 months ahead per opening, with 95% CI |\n\n---\n\n## Architecture\n\nSee [ARCHITECTURE.md](ARCHITECTURE.md) for full module specifications, data schemas, and mathematical derivations.\n\nSee [SCHEMAS.md](SCHEMAS.md) for artifact-level contracts and [READING_OPENCAST.md](READING_OPENCAST.md) for the release-facing interpretation guide.\n\n```mermaid\n---\nconfig:\n  layout: dagre\n---\nflowchart TB\n subgraph CATALOG[\"1. Catalog and Selection\"]\n  direction TB\n    CATALOGCSV[\"openings_catalog.csv\"]\n    FLAGS[\"compute_selection_flags.py\"]\n    BATCH[\"temp_bootstrap_openings.py\\n(offset/limit batches)\"]\n  end\n subgraph FETCH[\"2. Fetch\"]\n  direction TB\n    LICHESS[\"Lichess Explorer API\"]\n    RUST[\"fetcher v0.2.x\"]\n    EARLY[\"early-stop on below-min ratio\"]\n    RAW[\"data/raw/{group}/{ECO}.json\\nmonths + skipped metadata\"]\n  end\n subgraph INGEST[\"3. Ingest\"]\n    direction LR\n    CLEAN[\"ingest.py\"]\n    HIST[\"processed openings_ts.csv\"]\n  end\n subgraph ANALYSE[\"4. Analyse\"]\n    direction TB\n    TS[\"timeseries.py\\nARIMA/Holt-Winters\"]\n    DELTA[\"engine_delta.py\"]\n  end\n subgraph OUTPUTS[\"5. Output Artifacts\"]\n    direction TB\n    FOUT[\"output/forecasts.csv\"]\n    DOUT[\"output/engine_delta.csv\"]\n    SEL[\"data/selection_flags.csv\"]\n  end\n subgraph PUBLISH[\"6. Publish\"]\n    direction TB\n    REPORT[\"report.py -\u003e findings.md/json\"]\n    DASH[\"visualizer.py -\u003e dashboard site\"]\n    PAGE[\"GitHub Pages\"]\n  end\n\n  RUNNER[\"main.py\"] --\u003e CATALOG \u0026 FETCH \u0026 INGEST \u0026 ANALYSE \u0026 OUTPUTS \u0026 PUBLISH\n  CATALOGCSV --\u003e FLAGS --\u003e BATCH\n  BATCH --\u003e RUST\n  LICHESS --\u003e RUST --\u003e EARLY --\u003e RAW\n  BATCH --\u003e SEL\n    RAW --\u003e CLEAN\n  HIST --\u003e TS \u0026 DELTA\n  TS --\u003e FOUT\n    DELTA --\u003e DOUT\n  FOUT --\u003e REPORT \u0026 DASH\n  DOUT --\u003e REPORT \u0026 DASH\n  SEL --\u003e DASH\n    DASH --\u003e PAGE\n  REPORT --\u003e FINDINGS[\"findings/findings.md + findings.json\"]\n\n     LICHESS:::data\n   CATALOGCSV:::data\n   FLAGS:::ingest\n   BATCH:::ingest\n   RUST:::ingest\n   EARLY:::ingest\n     RAW:::data\n     CLEAN:::ingest\n     HIST:::data\n   TS:::analyse\n     DELTA:::analyse\n     FOUT:::output\n     DOUT:::output\n   SEL:::output\n     DASH:::output\n   REPORT:::output\n  FINDINGS:::output\n    PAGE:::data\n     RUNNER:::runner\n    classDef runner  fill:#0f2742,stroke:#4a90d9,stroke-width:2px,color:#d9ecff\n    classDef ingest  fill:#13281d,stroke:#4caf82,stroke-width:1.5px,color:#daf5e4\n    classDef analyse fill:#241633,stroke:#9b72cf,stroke-width:1.5px,color:#f0e4ff\n    classDef output  fill:#332011,stroke:#e07b39,stroke-width:1.5px,color:#ffe9d6\n    classDef data    fill:#1b1f24,stroke:#6e7681,stroke-width:1px,color:#e6edf3\n```\n\n---\n\n## Requirements\n\n- **Rust** ≥ 1.75 (stable) — for the Lichess fetcher  \n- **Python** ≥ 3.11 — for analytics pipeline  \n- **Stockfish 16** — `sudo apt install stockfish` (or set `STOCKFISH_PATH`)  \n- **Lichess OAuth token** — free at https://lichess.org/account/oauth/token  \n- **Gemini API key** (optional) — set `GEMINI_API_KEY` in `.env` (for AI-generated findings)\n- **Groq API key** (optional) — set `GROQ_API_KEY` in `.env` (for fast LLM inference)\n\n---\n\n## Repository Structure\n\n```\nfetcher/              ← Rust binary (Lichess Explorer → JSON)\nsrc/\n  ingest.py           ← consolidated raw JSON → openings_ts.csv\n  select_openings.py  ← per-ECO tier classification → openings_catalog.csv\n  timeseries.py       ← ARIMA (Tier 1) + Holt-Winters (Tier 2) forecasting\n  engine_delta.py     ← Stockfish centipawn → win probability delta\n  report.py           ← findings/findings.md + findings/findings.json\n  visualizer.py       ← multi-page static site generator\n  assets/\n    shared.css        ← design tokens + component styles\n    nav.js            ← active-link highlight\nscripts/\n  build_catalog.py          ← build/refresh full ECO catalog\n  compute_selection_flags.py ← tier flags + pruning\n  clean_raw_json.py         ← normalize/reformat consolidated raw JSON files\n  temp_bootstrap_openings.py ← batch bootstrap fetch with offset/limit and tracking\n  migrate_raw.py            ← legacy raw format migration helper\ndata/\n  raw/                ← grouped ECO JSON files at raw/{A-E}/{ECO}.json (gitignored)\n  processed/          ← openings_ts.csv\n  openings_catalog.csv ← ECO tier flags (is_tracked_core, model_tier, …)\n  opening_lines.json  ← curated canonical lines per ECO for interactive board playback\n  selection_flags.csv ← per-ECO coverage/tier diagnostics\n  output/\n    move_stats.csv    ← per-move monthly stats (generated locally/CI, not versioned)\n    forecasts.csv     ← ARIMA / HW forecasts with confidence intervals\n    engine_delta.csv  ← centipawn vs human win rate delta\n    long_tail_stats.csv ← Tier-3 coverage and descriptive opening stats\n    dashboard/        ← multi-page static site (GitHub Pages root)\n      index.html      ← overview + 3 panels\n      openings.html   ← sortable table of all ECOs\n      families.html   ← ECO family (A–E) summary\n      opening.html    ← single per-opening template with tier badge (use ?eco=B20)\n      assets/         ← shared.css, nav.js, openings_data.json, opening_lines.json\nfindings/\n  findings.md         ← narrative findings report\n  findings.json       ← structured findings payload\n  narratives.json     ← per-opening generated narrative cache\nopenings.json         ← seed opening definitions (legacy bootstrap input)\nmain.py               ← pipeline orchestrator\n```\n\n---\n\n## CI / Automation Notes\n\n- `.github/workflows/update.yml` fetches missing months incrementally, then runs a full recomputation by clearing generated artifacts and executing `AUTO_FETCH_MISSING_DATA=false python main.py`.\n- Processing commits include refreshed `data/processed/openings_ts.csv`, `data/output/*.csv`, dashboard pages, and `findings/` artifacts.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoeusyk%2Fopencast","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcoeusyk%2Fopencast","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoeusyk%2Fopencast/lists"}