{"id":51913142,"url":"https://github.com/stiles/datasiphon","last_synced_at":"2026-07-27T07:30:41.697Z","repository":{"id":367773062,"uuid":"1282235311","full_name":"stiles/datasiphon","owner":"stiles","description":"Turn captured browser traffic (HAR or cURL) into clean, reproducible data harvests. A last-mile tool for finding and harvesting hidden APIs, built for data journalists.","archived":false,"fork":false,"pushed_at":"2026-06-27T14:40:45.000Z","size":94,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-27T16:17:19.804Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stiles.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-27T13:58:15.000Z","updated_at":"2026-06-27T14:40:48.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/stiles/datasiphon","commit_stats":null,"previous_names":["stiles/datasiphon"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/stiles/datasiphon","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stiles%2Fdatasiphon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stiles%2Fdatasiphon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stiles%2Fdatasiphon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stiles%2Fdatasiphon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stiles","download_url":"https://codeload.github.com/stiles/datasiphon/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stiles%2Fdatasiphon/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35942774,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-27T02:00:06.776Z","response_time":101,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-27T07:30:40.985Z","updated_at":"2026-07-27T07:30:41.684Z","avatar_url":"https://github.com/stiles.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# datasiphon\n\nTurn captured browser traffic into clean, reproducible data harvests.\n\nModern sites load their data from \"hidden\" JSON APIs that the page never fully\nshows. Your browser already records every one of those requests. `datasiphon`\ntakes that capture (a HAR file or a single \"Copy as cURL\" string), finds the\nrequest that carries the data, and turns it into a repeatable harvest that\noutputs a spreadsheet plus a recipe other people can re-run and verify.\n\nIt is built for newsroom developers and data journalists. The goal is not to\ndefeat anti-bot systems but to close the last mile between \"I found the API in\nthe Network tab\" and \"I have clean, documented data.\"\n\n## Why a recipe\n\nEvery harvest produces a `recipe.yaml`: a human-readable record of the endpoint,\nheaders, pagination, and field mapping used to collect the data. It is the\n\"show your work\" artifact: version it, share it, and let an editor or another\nnewsroom reproduce your collection exactly.\n\n## Status\n\nThe full loop works end to end:\n\n- `inspect` ranks the requests in a HAR file to find the likely data endpoint\n- `init` generates a `recipe.yaml` (format, record path, and pagination\n  auto-detected, with an optional live probe)\n- `run` harvests every page and writes CSV, SQLite, and a manifest\n- `peek` normalizes a cURL command piped on stdin\n\nFormats supported: JSON, XML, and HTML tables.\nPagination supported: offset, page number, cursor, \"next\" field/URL, and the\nHTTP `Link` header (`rel=\"next\"`).\n\n## Install\n\n```bash\nnpm install\n```\n\n## Usage\n\n```bash\n# 1. Which request in this capture is the data?\nnpm run siphon -- inspect examples/sample.har\n\n# 2. Turn the top-ranked request into a recipe (or use --pick N from inspect)\nnpm run siphon -- init examples/sample.har -o recipe.yaml\n\n# 3. Review recipe.yaml, then harvest every page\nnpm run siphon -- run recipe.yaml\n\n# Resume an interrupted harvest where it left off\nnpm run siphon -- run recipe.yaml --resume\n\n# Quick check of a \"Copy as cURL\" string\npbpaste | npm run siphon -- peek\n```\n\nAfter `npm run build`, the `datasiphon` binary runs the same commands directly.\n\n### When the HAR has no response bodies\n\nBrowsers often save HARs without response bodies (\"Save all as HAR\" vs \"...with\ncontent\"). Then `inspect` can only rank by URL signals, and `init` can't see\nthe data shape. Use `--probe` so `init` fetches the endpoint once to detect the\nformat, record path, and pagination:\n\n```bash\nnpm run siphon -- init capture.har --pick 1 --probe\n```\n\n`init` probes automatically when it has no record path and you haven't passed\n`--no-probe`. Probing only fetches GET requests.\n\n### When the endpoint is a config that points elsewhere\n\nSome sites serve a configuration object instead of the data: a chart or\ndashboard loads a small JSON file describing how to render, and that file names\nthe real data file in a `dataUrl` (or `dataFileName`) field. CDC's COVE\nvisualizations work this way.\n\nWhen `--probe` fetches a response like that and finds no record array of its\nown, it follows the `dataUrl` (resolving relative paths against the config URL)\nand builds the recipe from the file that actually holds the records:\n\n```bash\nnpm run siphon -- init cdc-config.curl --probe\n# Probing https://www.cdc.gov/measles/weekly-cases-chart.json ...\n#   config response points elsewhere; following dataUrl to .../MeaslesCasesWeekly.json\n# source: GET https://www.cdc.gov/wcms/vizdata/measles/MeaslesCasesWeekly.json\n```\n\nIt follows at most three hops and stops if a `dataUrl` loops back on itself.\n\n### Formats\n\n`init` detects whether a response is JSON, XML, or HTML and writes the right\n`rows` block:\n\n- JSON / XML: `rows.path` is a small JSONPath (`$.results`, `$.records.record`)\n- HTML: `rows.selector` is a CSS selector for the repeating row; add\n  `rows.fields` to map named columns to per-row selectors, or let it fall back\n  to table cells (`c1`, `c2`, ...).\n\nFor HTML, `--probe` suggests a `rows.selector` by finding the largest data\ntable on the page (or, failing that, the most-repeated element). When the table\nhas a header row it also pre-fills `rows.fields` from the column headers. The\nsuggestion is a starting point: review it before harvesting.\n\n### Secrets\n\nBy default `init` keeps headers like `Authorization` and `Cookie` (often\nrequired) and warns you to treat the recipe as a secret. Use `--redact` to move\nthem out of the recipe into environment variables:\n\n```bash\nnpm run siphon -- init capture.har --probe --redact\n# recipe now has: Authorization: ${DATASIPHON_AUTHORIZATION}\nDATASIPHON_AUTHORIZATION='Bearer ...' npm run siphon -- run recipe.yaml\n```\n\n### Try the whole loop offline\n\n```bash\nnode examples/mock-server.mjs        # test API on :8787, leave running\n# in another shell:\nnpm run siphon -- run examples/local-recipe.yaml\n```\n\nThe mock server exposes endpoints for every supported pattern (offset, Link\nheader, cursor, XML, HTML, and an auth-protected route) so you can try\n`init --probe` and `run` against each one.\n\n### Fixtures from real APIs\n\nReal two-page captures used as regression fixtures, each exercising a different\nreal-world shape:\n\n| Fixture | Source | Exercises |\n| --- | --- | --- |\n| `socrata_nyc311.har` | NYC 311 (Socrata) | top-level array, `$offset`/`$limit` |\n| `socrata_la_crime.har` | LAPD crime, data.lacity.org (Socrata) | same, LA data |\n| `arcgis_la_streetlights.har` | LA Bureau of Street Lighting (ArcGIS) | `$.features`/`attributes`, `resultOffset`, `exceededTransferLimit` |\n\nRegenerate them (or capture new fixtures) with the bundled script, which fetches\nlive pages and writes a HAR with response bodies:\n\n```bash\n# Socrata\nnode examples/capture.mjs \\\n  --url 'https://data.lacity.org/resource/2nrs-mtv8.json?$limit=5\u0026$offset=0' \\\n  --out socrata_la_crime --pages 2 --offset-param '$offset' --limit 5\n\n# ArcGIS FeatureServer / MapServer\nnode examples/capture.mjs \\\n  --url 'https://maps.lacity.org/lahub/rest/services/Bureau_of_Street_Lighting/MapServer/0/query?where=1=1\u0026outFields=*\u0026returnGeometry=false\u0026resultRecordCount=3\u0026resultOffset=0\u0026f=json' \\\n  --out arcgis_la_streetlights --pages 2 --offset-param 'resultOffset' --limit 3\n```\n\nA `run` produces `\u003cbasename\u003e.csv`, `\u003cbasename\u003e.sqlite`, and\n`\u003cbasename\u003e.manifest.json`. The manifest records the source, pagination type,\npage and row counts, columns (with their inferred SQLite types), and any\nwarnings: your reproducibility receipt.\n\nThe SQLite table infers a type per column (`INTEGER`, `REAL`, or `TEXT`) so\nnumbers come back as numbers. Codes that only look numeric are kept as text:\nvalues with a leading zero (ZIP/FIPS codes) and very long digit strings (IDs,\nphone numbers) stay `TEXT` so they aren't silently mangled.\n\n## How to capture traffic\n\n`datasiphon` does not intercept traffic itself; your browser already does it\nbetter than a new tool could. Two ways to get a capture:\n\n1. **HAR** (many requests): DevTools \u003e Network tab \u003e right-click \u003e \"Save all as\n   HAR with content\". Best for letting `inspect` find the data endpoint.\n2. **cURL** (one request): DevTools \u003e Network tab \u003e right-click a request \u003e\n   Copy \u003e \"Copy as cURL\". Best when you already know which request you want.\n\n## Roadmap\n\n- [x] HAR and cURL parsing\n- [x] Request ranking (`inspect`)\n- [x] Recipe generation (`init`), with live `--probe`\n- [x] Pagination: offset, page, cursor, next field, Link header\n- [x] ArcGIS support: `resultOffset`, `exceededTransferLimit`, `attributes` unwrap\n- [x] Harvest engine: polite replay, retries, resume (`run`)\n- [x] Flatten nested JSON/XML to CSV + SQLite\n- [x] XML and HTML-table record extraction\n- [x] Secret redaction (`--redact`) + env-var substitution on `run`\n- [x] Harvest manifest (source, pagination, row counts, warnings)\n- [x] Follow a config's `dataUrl` (CDC-style viz endpoints point elsewhere)\n- [x] Auto-suggest an HTML row selector during `--probe`\n- [x] Type inference for SQLite columns (INTEGER/REAL/TEXT)\n\n## Not in scope\n\nBot/captcha evasion, mobile app interception, and headless-browser capture.\nUse `mitmproxy` or Playwright for those; `datasiphon` focuses on the last mile.\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstiles%2Fdatasiphon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstiles%2Fdatasiphon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstiles%2Fdatasiphon/lists"}