{"id":51485560,"url":"https://github.com/edygert/bhindex","last_synced_at":"2026-07-07T06:02:02.675Z","repository":{"id":366354212,"uuid":"1275977840","full_name":"edygert/bhindex","owner":"edygert","description":"Local-first harvester and index for Black Hat conference session metadata (Wayback-sourced, SQLite + FTS5, TUI/FastAPI-ready)","archived":false,"fork":false,"pushed_at":"2026-06-21T12:09:57.000Z","size":73,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-06-21T14:25:59.369Z","etag":null,"topics":["black-hat","full-text-search","osint","python","sqlite","tui","wayback-machine"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/edygert.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-21T11:41:23.000Z","updated_at":"2026-06-21T12:10:01.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/edygert/bhindex","commit_stats":null,"previous_names":["edygert/bhindex"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/edygert/bhindex","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edygert%2Fbhindex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edygert%2Fbhindex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edygert%2Fbhindex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edygert%2Fbhindex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/edygert","download_url":"https://codeload.github.com/edygert/bhindex/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edygert%2Fbhindex/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35216572,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-07T02:00:07.222Z","response_time":90,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["black-hat","full-text-search","osint","python","sqlite","tui","wayback-machine"],"created_at":"2026-07-07T06:02:01.611Z","updated_at":"2026-07-07T06:02:02.654Z","avatar_url":"https://github.com/edygert.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# bhindex\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n[![Python 3.12+](https://img.shields.io/badge/python-3.12%2B-blue.svg)](https://www.python.org/)\n\nLocal-first harvester and index for **Black Hat** conference session metadata. It crawls event\nmetadata into a local SQLite database and gives you full-text search over it — **metadata only; it\nnever downloads presentation files or other binaries.**\n\nThis is **Phase 1 (backend + CLI)**. A TUI and a FastAPI layer are planned to wrap the same service\nlayer later (see *Architecture*). Scope is intentionally **2016+** (see *Coverage*).\n\n## Why the Wayback Machine\n\n`blackhat.com` is fronted by a Cloudflare managed challenge that returns `403` to non-browser HTTP\nclients on datacenter IPs. Rather than fight it, bhindex harvests from the **Wayback Machine**, which\nis not behind Cloudflare and works from any network (including CI). Because past Black Hat events are\nimmutable, the archive already holds essentially all historical metadata.\n\n## How harvesting works\n\n- **Recent events (2016+)** publish a structured JSON feed at\n  `…/\u003cedition\u003e/briefings/schedule/sessions.json` (the schedule page itself is a Handlebars template\n  rendered from this feed). bhindex parses the feed → events, sessions, speakers, abstracts, and\n  material links.\n- **2016 \u0026 2017 materials** are *not* in the feed. The files live under\n  `blackhat.com/docs/\u003cedition\u003e/…/\u003cedition\u003e-\u003cPresenter\u003e-\u003cTitle\u003e.pdf`. bhindex enumerates them via the\n  Wayback CDX API and attaches each to its session by matching the filename to a speaker surname\n  (primary) plus title-token overlap (fallback). Non-session files (surveys, letters) are skipped.\n- **2018+ materials** come straight from the feed (`bh_files`, pointing at `i.blackhat.com`).\n- **Manual / offline**: you can save a `sessions.json` (or any page) yourself and `ingest-file` it.\n\nHarvesting fetches HTML/JSON only and records material **URLs** as metadata. There is no code path\nfrom harvesting to downloading a binary.\n\n## Install\n\nRequires Python 3.12+ and [uv](https://docs.astral.sh/uv/).\n\n```bash\nuv sync                 # install runtime + dev deps\nuv run bhindex --help\n```\n\n## Usage\n\n```bash\nuv run bhindex init-db                      # create the database + FTS index\nuv run bhindex harvest us-24 eu-23 asia-24  # harvest one or more editions (metadata only)\nuv run bhindex harvest us-17                # 2017: materials backfilled from /docs automatically\nuv run bhindex harvest us-24 --refresh      # re-download, ignoring the local cache\n# harvest shows a live progress monitor and a per-event validation report (anomalies / missing data).\nuv run bhindex stats                        # row counts + per-source/per-event coverage\nuv run bhindex events                       # list harvested events\nuv run bhindex search \"kerne\"               # substring search (trigram); prints a #id per result\nuv run bhindex search \"malware\" --all       # every match (default shows 50; -n to change)\nuv run bhindex search \"ssh\" --json | jq .   # full per-hit metadata as a JSON array (for piping)\nuv run bhindex show 1234                     # full detail for one session (speakers, abstract, links)\nuv run bhindex show 1234 --json             # that session's full metadata as a JSON object\n\n# offline / manually-saved page:\nuv run bhindex ingest-file ./sessions.json --url https://www.blackhat.com/us-24/briefings/schedule/sessions.json\n```\n\nEditions are written as they appear in the URL: `us-24`, `eu-23`, `asia-24`.\n\n`search` is substring-based (FTS5 trigram tokenizer over title/abstract/speakers/track/material\ntitles), so `kerne` finds `Kernel`; terms shorter than 3 characters fall back to a `LIKE` scan. Human\noutput prints `#id` rows on stdout with the match-count summary on stderr (so it pipes cleanly);\n`--json` emits full per-hit metadata (`search` → JSON array, `show` → JSON object) on stdout.\n\nData lives under `~/.local/share/bhindex/` (override with `--data-dir` or `BHINDEX_DATA_DIR`):\n`bhindex.sqlite3`, `snapshots/` (raw HTML/JSON captures for debugging + parser tests), `cache/`, and\n`bhindex.log`.\n\n**Caching.** Wayback responses (feeds + CDX enumerations) are immutable, so each is cached under\n`cache/` and reused on later runs — re-processing after a parser change, and re-running a sweep, no\nlonger re-download. Cached editions are served from disk (no network, no delay); only new/uncached\neditions are fetched, so a re-run is naturally incremental. Use `--refresh` to ignore the cache and\npull fresh captures (e.g. for a just-finalized event).\n\n**Politeness.** The Wayback Machine rate-limits aggressively, so bhindex paces requests with a\n**5 s default delay + jitter**, a single continuously-paced client across a whole multi-edition run,\nand exponential backoff that honors `Retry-After` on `429`/`5xx` (throttling is surfaced live in the\nprogress monitor). Tune via `BHINDEX_REQUEST_DELAY`, `BHINDEX_MAX_RETRIES`, `BHINDEX_USER_AGENT`\n(or `~/.config/bhindex/config.toml`). A full 2016–2025 sweep (30 editions) takes ~10–12 min at the\ndefault delay — slower is friendlier.\n\n## Architecture\n\nLayered backend; the dependency rule points inward. The CLI (and a future TUI/FastAPI) call **only**\nthe service layer — never repositories, parsers, or fetchers directly.\n\n```\ncli ─► services ─► parsers          (recent feed parser, /docs materials matcher)\n            │   └─► adapters         (HttpClient, WaybackFetcher, FileFetcher)\n            └─► storage              (sqlite3 + FTS5, repositories, snapshots)\n   everything ─► core / dto          (config, errors, urls, models; pydantic contracts)\n```\n\n- `ServiceContainer` is the DI root; `HarvestService`, `SearchService`, `StatsService` are the public API.\n- Services are **synchronous** and FastAPI-friendly: a future web layer constructs the same container\n  and calls the same methods; a TUI drives them from background workers. No core logic changes needed.\n- Parsers are pure (HTML/JSON + base URL → DTOs); all network I/O lives in adapters.\n\n## Coverage (what actually works)\n\n| Years | Sessions + speakers + abstracts | Materials |\n|---|---|---|\n| 2018–2025 (us/eu/asia) | ✅ feed | ✅ feed (`i.blackhat.com`) |\n| 2016–2017 | ✅ feed | ✅ backfilled from `blackhat.com/docs/` |\n| ≤2015 | ❌ no JSON feed | ❌ |\n\nPre-2016 legacy `bh-media-archives` HTML is intentionally **out of scope** — its markup is too\nheterogeneous across years to parse reliably, and is not currently needed.\n\n## Development\n\n```bash\nuv run pytest        # unit + integration tests (fixtures are real, trimmed captures)\nuv run ruff check .  # lint\n```\n\nTests run fully offline against fixture data in `tests/fixtures/`. The integration suite pins the\nPhase-1 invariant that harvesting writes only metadata + HTML snapshots, never binary files.\n\n## License\n\n[MIT](LICENSE) © 2026 Evan H. Dygert.\n\nbhindex indexes only **metadata** (titles, abstracts, speakers, and links). It does not redistribute\nBlack Hat presentation files; all material links point back to their original sources. Respect the\nsource sites' terms of use and the Internet Archive's access policies when harvesting.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fedygert%2Fbhindex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fedygert%2Fbhindex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fedygert%2Fbhindex/lists"}