https://github.com/buger/source-bias-atlas
https://github.com/buger/source-bias-atlas
Last synced: 8 days ago
JSON representation
- Host: GitHub
- URL: https://github.com/buger/source-bias-atlas
- Owner: buger
- License: mit
- Created: 2026-05-08T09:57:11.000Z (about 1 month ago)
- Default Branch: main
- Last Pushed: 2026-05-24T15:23:00.000Z (18 days ago)
- Last Synced: 2026-05-24T17:23:54.875Z (18 days ago)
- Language: Python
- Size: 19.7 MB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Source Bias Atlas
An interactive cluster map of [daily.dev](https://daily.dev) content sources, grouped by stylistic personality (hype vs. depth, generalist vs. niche, cadence, etc.). Pan, zoom, search, swap views, drill into a source, compare two sources side-by-side, or build a "reading diet" by tuning sliders.
**Live:** https://buger.github.io/source-bias-atlas/
Sibling project: **[Source Originality Score](https://buger.github.io/source-originality-score/)** — who breaks each story first.
## What's inside
```
crawler/ # daily.dev source + post crawler (Python, DuckDB)
features/ # per-source feature extraction
clustering/ # UMAP + KMeans + per-feature views
probe/ # API exploration scripts (kept for reproducibility)
web/ # Next.js 14 app (the static site)
```
The `web/public/` directory holds the committed artifacts that the site reads at build time:
- `atlas-summary.json` — map view + per-source summaries
- `clusters.json` — cluster metadata (labels, colors)
- `sources/.json` — full per-source detail (~1,360 files)
## Deploy
Push to `main`. The GitHub Actions workflow at `.github/workflows/deploy.yml` builds with `next export` and publishes `web/out/` to GitHub Pages.
## Run locally
```bash
cd web
pnpm install
pnpm dev # http://localhost:3000
```
## Rebuild data
```bash
# 1. crawl daily.dev for sources + recent posts
DAILYDEV_TOKEN=… python crawler/run.py
# 2. extract features per source
python features/build.py
# 3. cluster + lay out
python clustering/run.py
# 4. dump JSON artifacts into web/public/
python clustering/export_web.py
```
## Caveats
- Feature definitions are deliberate but not authoritative — "hype" is a synthetic score, not a moral judgment.
- The atlas is a snapshot in time; daily.dev posts churn.
- Tags come straight from daily.dev; we don't normalize.
## License
MIT — see `LICENSE`.