{"id":51344165,"url":"https://github.com/nilicule/codewall","last_synced_at":"2026-07-02T10:03:55.983Z","repository":{"id":367265359,"uuid":"1280010264","full_name":"nilicule/codewall","owner":"nilicule","description":"A self-contained Flask app that shows live GitHub activity across an organisation","archived":false,"fork":false,"pushed_at":"2026-06-25T09:01:55.000Z","size":76,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-25T09:19:08.302Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nilicule.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-25T07:32:31.000Z","updated_at":"2026-06-25T09:01:59.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/nilicule/codewall","commit_stats":null,"previous_names":["nilicule/codewall"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/nilicule/codewall","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nilicule%2Fcodewall","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nilicule%2Fcodewall/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nilicule%2Fcodewall/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nilicule%2Fcodewall/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nilicule","download_url":"https://codeload.github.com/nilicule/codewall/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nilicule%2Fcodewall/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35042011,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-02T02:00:06.368Z","response_time":173,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-02T10:03:55.150Z","updated_at":"2026-07-02T10:03:55.972Z","avatar_url":"https://github.com/nilicule.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NET2GRID Activity Wall\n\nA self-contained Flask app that shows live GitHub activity across the NET2GRID\norganisation: a cinematic spiral constellation hero, a live commit/PR stream, a\n\"who is working where\" roster, a per-repo activity breakdown, a 90-day\ncontribution heatmap, and an \"org pulse\" ECG that beats with every event. Viewers\nlog in once and watch the org work in real time.\n\nThe UI is the approved prototype (`dashboard.html`), served as the dashboard\ntemplate with its mock data generator swapped for `fetch()` calls to the Flask\nAPI. Everything runs in one container with no external services.\n\n## Quick start (local, no secrets)\n\nWith no GitHub token set, the app serves mock data so the whole UI runs with\nzero infrastructure.\n\n```bash\nuv sync\nDEV_AUTH_BYPASS=1 uv run flask --app app run --port 5008\n# open http://127.0.0.1:5008\n```\n\n`DEV_AUTH_BYPASS=1` skips OAuth and treats every request as an org member. It is\nclearly logged as insecure and must never be set in production.\n\nRun the tests:\n\n```bash\nuv run pytest tests/test_snapshot.py tests/test_harvester.py tests/test_auth.py  # units\nuv run playwright install chromium                                               # one-time\nuv run pytest tests/test_smoke.py                                                # browser smoke test\nuv run pytest                                                                    # everything\n```\n\nThe suite never touches the live API (mock data, fake GraphQL fixtures).\n\n## Running against real GitHub\n\nConfiguration lives in a `.env` file in the project root (loaded automatically).\nStart from the template:\n\n```bash\ncp .env.example .env\n```\n\n**1. Harvest token.** Create a read-only GitHub token for reading org activity.\nA fine-grained PAT with read access to the org's repositories (Contents,\nMetadata, Pull requests) is enough; a GitHub App installation token also works.\nIt is used server-side only and never sent to the client.\n\n**2. Access key (viewer gate).** The simplest gate is one shared secret. Generate\none:\n\n```bash\npython -c \"import secrets; print(secrets.token_urlsafe(32))\"\n```\n\n(Prefer per-user Google Workspace login instead? See \"Gating who can view\" below.)\n\n**3. Session signing key.**\n\n```bash\npython -c \"import secrets; print(secrets.token_hex(32))\"\n```\n\n**4. Fill in `.env`:**\n\n```ini\nGITHUB_TOKEN=github_pat_...          # from step 1\nGITHUB_ORG=NET2GRID\nACCESS_TOKEN=...                     # from step 2\nSECRET_KEY=...                       # from step 3\n```\n\n**5. Boot:**\n\n```bash\nuv sync\nuv run flask --app app run --port 5008      # dev server\n# or, for production (single worker, see below):\nuv run gunicorn -w 1 --threads 8 -b 0.0.0.0:5008 app:app\n```\n\nOpen http://127.0.0.1:5008 and enter the access key once. The first full harvest\nscans the whole window and can take a couple of minutes across hundreds of repos;\nthe dashboard shows zeros until it completes, then refreshes incrementally every\n`REFRESH_SECONDS`. Serve behind HTTPS in production.\n\n### Gating who can view (pick one)\n\nThe dashboard and all `/api/*` routes require a session; only the login route and\nstatic assets are public. The session is a Flask signed cookie (`SECRET_KEY`), so\nthere is no session store and we stay single-container. Three gates, in precedence\norder:\n\n1. **Shared secret (simplest).** Set `ACCESS_TOKEN` to a long random string.\n   Viewers open `/login`, enter the key once, and get a signed-cookie session. No\n   OAuth app, no per-user identity: anyone with the key can view. Good for\n   an internal wall, especially behind HTTPS.\n2. **Google Workspace OAuth.** Set `GOOGLE_CLIENT_ID` / `GOOGLE_CLIENT_SECRET`\n   (OAuth client with callback `https://\u003cyour-host\u003e/callback`). Google login -\u003e\n   the signed `id_token` is verified and the account must be a verified member of\n   `ALLOWED_EMAIL_DOMAIN` (default `net2grid.com`) -\u003e session. Per-user identity\n   and automatic domain-only enforcement.\n3. **`DEV_AUTH_BYPASS=1`.** Skips auth entirely for local dev. Insecure; never in\n   production.\n\nIf none are set, the app stays locked. Note the harvest `GITHUB_TOKEN` is a\nseparate, server-side, read-only credential (repo Contents/Metadata/Pull requests\nread) and is never sent to the client; with the shared-secret gate it needs no\n`read:org` scope. You can also drop app auth entirely and put the container behind\nan identity-aware proxy or private network (Cloudflare Access, Tailscale, VPN).\n\n### Setting up Google Workspace login\n\nOne-time setup in the [Google Cloud Console](https://console.cloud.google.com/),\nsigned in with a `net2grid.com` account:\n\n1. **Project** — create (or reuse) a project, e.g. `codewall-auth`, and make sure\n   it is selected.\n2. **OAuth consent screen** (APIs \u0026 Services → OAuth consent screen) — set\n   **User type: Internal**. This restricts sign-in to `net2grid.com` Workspace\n   accounts and skips Google's app-verification process. Fill in the app name and\n   support/developer emails. The `openid`, `email`, and `profile` scopes are basic\n   and need not be added explicitly.\n3. **Credentials** (APIs \u0026 Services → Credentials) → **Create Credentials → OAuth\n   client ID** → **Application type: Web application**. Under **Authorized redirect\n   URIs**, add the callback URL — it must match exactly what the app builds,\n   `\u003cscheme\u003e://\u003chost\u003e\u003cURL_PREFIX\u003e/callback`:\n   - root: `https://your-host.example.com/callback`\n   - sub-path (`URL_PREFIX=/codewall`): `https://your-host.example.com/codewall/callback`\n   - local: `http://localhost:5008/callback`\n\n   Add every host you use (prod + localhost). The match is exact — scheme, host,\n   and path all count, with no trailing slash.\n4. **Copy the Client ID and Client secret** into the environment as\n   `GOOGLE_CLIENT_ID` / `GOOGLE_CLIENT_SECRET` (plus a strong `SECRET_KEY`, and\n   optionally `ALLOWED_EMAIL_DOMAIN` if not `net2grid.com`). Leave `ACCESS_TOKEN`\n   and `DEV_AUTH_BYPASS` unset — they outrank the Google gate.\n\nViewers then land on a `/login` card with a **Continue with Google** button. The\n`redirect_uri` is derived from the `X-Forwarded-Proto`/`X-Forwarded-Host` headers\n(via `ProxyFix`), so a reverse proxy must set those correctly or Google returns\n`redirect_uri_mismatch`. No additional Google APIs need enabling — sign-in uses the\ndefault OpenID Connect endpoints.\n\n## Configuration\n\nAll configuration is via environment variables (see `.env.example`). A `.env`\nfile in the project root is loaded automatically in local dev. Live versus mock\nis decided solely by whether `GITHUB_TOKEN` is non-empty. Note that values which\nare unset OR empty in the environment are filled from `.env`, so a stray empty\n`GITHUB_TOKEN=` export in your shell will not silently shadow the real token and\nforce mock. To deliberately run mock while a token sits in `.env`, comment that\nline out, or run with `N2G_SKIP_DOTENV=1` and an empty `GITHUB_TOKEN`.\n\n| Variable | Default | Purpose |\n| --- | --- | --- |\n| `GITHUB_TOKEN` | (empty) | Server-side token. Empty = serve mock data. |\n| `GITHUB_ORG` | `NET2GRID` | Organisation to harvest. |\n| `WINDOW_DAYS` | `90` | Rolling activity window. |\n| `REFRESH_SECONDS` | `120` | Seconds between real harvests. |\n| `MOCK_REFRESH_SECONDS` | `3` | Tick interval in mock mode. |\n| `SECRET_KEY` | dev value | Signs the session cookie. Set a strong value in prod. |\n| `ACCESS_TOKEN` | (empty) | Shared access key for the simple viewer gate. |\n| `GOOGLE_CLIENT_ID` | (empty) | Google OAuth client id (Workspace viewer gate). |\n| `GOOGLE_CLIENT_SECRET` | (empty) | Google OAuth client secret. |\n| `ALLOWED_EMAIL_DOMAIN` | `net2grid.com` | Workspace domain allowed to sign in. |\n| `DEV_AUTH_BYPASS` | `0` | `1` = skip the viewer gate entirely. Local only, insecure. |\n| `CACHE_PERSIST_PATH` | (empty) | Path to a SQLite file. Empty = pure in-memory. |\n| `URL_PREFIX` | (empty) | Sub-path mount when behind a proxy (e.g. `/codewall`). |\n| `N2G_SKIP_DOTENV` | (empty) | `1` = ignore `.env` (used by tests to force mock). |\n\n## How the cache and background refresh work\n\nThis dashboard renders a rolling 90-day window of org activity that changes only\nevery few minutes. That is a cache, not a system of record, so no database is\nneeded.\n\n* The harvested data lives in a module-level Python object (`n2g.snapshot.Store`)\n  guarded by a `threading.Lock`.\n* A single daemon `threading.Thread` (`n2g.harvester.Harvester`), started once on\n  boot, runs the loop: harvest -\u003e shape -\u003e store -\u003e sleep `REFRESH_SECONDS` -\u003e\n  repeat. It is the only code that talks to GitHub.\n* Request handlers never call GitHub. They take a quick copy of the snapshot\n  under the lock and return it. The lock is never held across network calls.\n* Harvesting uses the GitHub GraphQL API v4 (not REST) and batches 25 repos per\n  page so we stay efficient across hundreds of repos. It is incremental: commit\n  history is fetched `since` the last harvest and PRs are pulled most-recently-\n  updated first, so refreshes pull only what changed. The rolling window is\n  trimmed on every cycle.\n* GraphQL gives 5000 points/hour. The remaining budget is logged after each\n  refresh, and the loop backs off (sleeps longer) when the budget runs low.\n\nThe frontend polls the JSON API on an interval and animates events through the\nprototype's existing render functions (`firePulse`, the roster, the bars, the\nfeed). Real org activity is sparse (a few events every couple of minutes), so a\ncontinuous animator walks a rolling pool of recent events and loops back to the\nstart when it reaches the end, keeping the hero beams, the \"floor\" roster and\nthe Data Stream feed alive between refreshes; each poll appends genuinely-new\nevents to the pool and plays them next. The same stream drives the live reactions\nacross the wall: each animated event fires its hero beam, raises and flashes its\nauthor on the \"floor\" roster, flashes its repo in \"Where the work lands\", and\nblooms its day in the contribution-density strip, and pumps the \"org pulse\"\nvoicebox in the top-left (a KITT-style row of segmented LED columns whose loudness\ntracks recent event energy, tinted by event kind). So every panel reacts to real\nevents rather than sitting\non static totals; the window totals themselves live in a compact caption under the\nECG. The density strip also carries an ambient shimmer: every day breathes faintly,\nscaled by that day's real activity, so busy past weeks glimmer while quiet ones stay\ndark. The repo bars carry a perpetual sheen so they stay alive when totals hold\nsteady. The totals and bar widths stay authoritative from their endpoints. Polling,\nnot websockets, keeps this single-container and simple.\n\n## The single-worker requirement (important)\n\nThe state is in memory, so the app MUST run as ONE process. Run a single worker:\n\n```bash\ngunicorn -w 1 --threads 8 -b 0.0.0.0:5008 app:app\n```\n\nDo NOT run multiple Gunicorn workers. Each worker would get its own copy of the\ncache and its own background harvester, which would multiply GitHub API calls and\nserve inconsistent data depending on which worker handled a request. Scale via\nthe cache plus a CDN in front of the app, not via more processes. Use threads\n(`--threads`) for concurrency within the one worker. The background harvest\nthread starts on app import/boot, not per request.\n\n## JSON API\n\nAll endpoints read straight from the in-memory snapshot (no per-request GitHub\ncalls) and require an authenticated session. Only the login routes and static\nassets are public.\n\n| Endpoint | Returns |\n| --- | --- |\n| `GET /api/stats` | `{commits, repos_active, prs_open, prs_merged, people_active}` |\n| `GET /api/events/recent?limit=N` | reverse-chronological commits + PRs: `{ts, kind, dev, repo, message?}` |\n| `GET /api/contributors/active` | `{login, name, avatar, last_active, kind, repo}` per active contributor |\n| `GET /api/repos/top?limit=5` | `[{repo, count}]` |\n| `GET /api/heatmap` | per-day counts for the 90-day strip: `[{date, count}]` |\n| `GET /healthz` | liveness + snapshot `updated_at` |\n\n## Optional persistence\n\nBy default the cache is pure in-memory and a restart triggers a cold\nre-harvest. Set `CACHE_PERSIST_PATH` to persist the snapshot across restarts.\nThis is one local SQLite file (stdlib `sqlite3`, no server, no ORM): a single\ntable holding one JSON blob of the raw window, read once on boot and written\nafter each refresh. It does not violate the \"no external services\" rule.\n\nUse a **relative** path so the same value works in both environments:\n\n```ini\nCACHE_PERSIST_PATH=data/snapshot.sqlite\n```\n\nThe parent directory is created automatically. In local dev it resolves under\nthe project directory (`./data/snapshot.sqlite`, gitignored). In the container\nit resolves under `WORKDIR /app` (`/app/data/snapshot.sqlite`); the image\ndeclares `/app/data` as a volume, so mount one to keep the snapshot across\ncontainer recreations:\n\n```bash\ndocker run -p 5008:5008 --env-file .env -v n2g-cache:/app/data net2grid-wall\n```\n\nPersistence is also skipped in mock mode, and a path that cannot be opened (for\nexample an unwritable absolute path like `/data/...` on your laptop) disables\npersistence with a warning rather than crashing the app.\n\n## Behind a reverse proxy (sub-path)\n\nTo serve the wall under a sub-path (for example `https://host/codewall/`), set\n`URL_PREFIX=/codewall`. The app then mounts there: `url_for`, redirects and the\nlogin URL emit `/codewall/...`, and the frontend prefixes all of its API/login\nURLs with it (injected as `BASE`). It also honours `X-Forwarded-Proto/Host` so\nexternal URLs use the right scheme and host.\n\n`URL_PREFIX` works whether nginx strips the prefix or forwards it intact:\n\n```nginx\nlocation /codewall/ {\n    proxy_pass         http://app:5008/;   # trailing slash: nginx strips /codewall\n    proxy_set_header   Host              $host;\n    proxy_set_header   X-Forwarded-Proto $scheme;\n    proxy_set_header   X-Forwarded-Host  $host;\n    proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;\n}\n```\n\nRun the container with `-e URL_PREFIX=/codewall`. Served at the domain root,\nleave `URL_PREFIX` empty and nothing changes.\n\n## Docker\n\n```bash\ndocker build --network=host -t codewall .\ndocker run -p 5008:5008 --network host \\\n  -e GITHUB_TOKEN=... -e GITHUB_ORG=NET2GRID \\\n  -e ACCESS_TOKEN=... -e SECRET_KEY=... \\\n  net2grid-wall\n# or pass your filled-in .env directly:\ndocker run -p 5008:5008 --network host --env-file .env codewall\n```\n\nThe image runs exactly one Gunicorn worker with eight threads (see above).\n\n## Future: webhooks instead of polling\n\nToday the frontend polls and the harvester refreshes on an interval. To make\nupdates push-based you could register GitHub organisation webhooks (push,\npull_request) pointing at a new `POST /webhook` route that validates the\nsignature and applies the single event to the in-memory snapshot under the lock,\nthen have the frontend receive updates via Server-Sent Events. That would cut\nlatency and API usage, but it adds a public ingress endpoint and signature\nhandling, so it is intentionally out of scope here. The current polling model\nkeeps the system single-container and simple.\n\n## Project layout\n\n```\napp.py              entry point (flask --app app / gunicorn app:app)\nn2g/\n  __init__.py       Flask app factory + harvester boot\n  config.py         env-driven configuration\n  snapshot.py       in-memory Store (lock) + pure shaping logic\n  github.py         GraphQL v4 incremental harvester\n  mockdata.py       mock source used when no token is set\n  harvester.py      background refresh thread\n  auth.py           viewer gate: shared access key or Google Workspace OAuth\n  api.py            JSON API blueprint\n  persist.py        optional single-file SQLite persistence\ntemplates/dashboard.html   the prototype, wired to the API\ntests/              harvester + snapshot + auth units, Playwright smoke test\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnilicule%2Fcodewall","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnilicule%2Fcodewall","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnilicule%2Fcodewall/lists"}