{"id":51485598,"url":"https://github.com/edygert/dcindex","last_synced_at":"2026-07-07T06:02:11.897Z","repository":{"id":366380678,"uuid":"1276072452","full_name":"edygert/dcindex","owner":"edygert","description":"Local-first indexer for DEF CON conference schedule metadata (Outel MySQL dumps, DC26+). Metadata only; links displayed for manual access.","archived":false,"fork":false,"pushed_at":"2026-06-21T14:34:40.000Z","size":74,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-21T16:12:52.095Z","etag":null,"topics":["cli","conference","defcon","full-text-search","indexer","local-first","mysqldump","python","security","sqlite"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/edygert.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-21T14:05:03.000Z","updated_at":"2026-06-21T14:34:44.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/edygert/dcindex","commit_stats":null,"previous_names":["edygert/dcindex"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/edygert/dcindex","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edygert%2Fdcindex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edygert%2Fdcindex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edygert%2Fdcindex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edygert%2Fdcindex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/edygert","download_url":"https://codeload.github.com/edygert/dcindex/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/edygert%2Fdcindex/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35216572,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-07T02:00:07.222Z","response_time":90,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","conference","defcon","full-text-search","indexer","local-first","mysqldump","python","security","sqlite"],"created_at":"2026-07-07T06:02:11.106Z","updated_at":"2026-07-07T06:02:11.873Z","avatar_url":"https://github.com/edygert.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# dcindex\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n[![Python 3.12+](https://img.shields.io/badge/python-3.12%2B-blue.svg)](https://www.python.org/)\n\nLocal-first indexer for **DEF CON** conference schedule metadata. It ingests structured session data\nfrom the **\"Outel\" MySQL schedule dumps**, stores it in a local SQLite database, and gives you\nfull-text search over talks, villages, demo labs, workshops, trainings, and contests — **metadata\nonly; it never downloads presentation files, videos, or other assets.** Material/video/forum URLs are\nrecorded and displayed so you can open them yourself.\n\nThis is **Phase 1 (backend + CLI)**. The service layer is FastAPI-friendly so a TUI or web layer can\nwrap the same operations later.\n\n## Data source\n\nDEF CON schedules are republished in many formats by the unofficial **\"The One!\"** project at\n[`defcon.outel.org`](https://defcon.outel.org). Among them is a **MySQL dump** per year:\n\n```\nhttps://defcon.outel.org/defcon\u003cNN\u003e/dc\u003cNN\u003e_mysqldump.txt\n```\n\nDumps are available for **DC26–DC33 (2018–2025)**. These files are unofficial (\"not affiliated with\nDEF CON … use at your own risk\") and carry no stated license or redistribution restriction. dcindex\nindexes the **metadata only** and links back to original sources — it does not rebundle the dataset.\nPlease respect the source site and be considerate (dcindex paces and caches its single network call).\n\n## How it works\n\n- **`ingest-dump`** reads a local dump file (offline). It parses the `mysqldump` SQL directly — no\n  MySQL server needed.\n- **`ingest-url`** fetches a published dump **once**, saves it to a local cache\n  (`\u003cdata_dir\u003e/dumps/`), and ingests the cached copy. Re-running ingests from cache with **no network\n  call** (`--refresh` forces a re-download).\n- The dumps' core (`events` = talks, `speakers`) is stable across all years; auxiliary activity\n  tables drift year to year, so the mapper is tolerant — it maps whatever recognized tables a dump\n  contains (`demolabs`, `workshops`, `training`, `contests`, `villages`, and the DC32/33 consolidated\n  `pages`), records material/forum/video URLs found in columns **and** embedded in HTML descriptions,\n  and reports any unrecognized table rather than dropping it silently. `vendors` is excluded.\n\nIngest reads SQL text only and records material **URLs** as metadata. There is no code path from\ningest to downloading a binary asset.\n\n## Install\n\nRequires Python 3.12+ and [uv](https://docs.astral.sh/uv/).\n\n```bash\nuv sync\nuv run dcindex --help\n```\n\n## Usage\n\n```bash\nuv run dcindex init-db                       # create the database + FTS index\nuv run dcindex ingest-url --edition dc33     # download (once, cached) + ingest DC33\nuv run dcindex ingest-url --edition dc32     # …and more years\nuv run dcindex ingest-dump ./dc30_mysqldump.txt --edition dc30   # offline, from a local file\n\nuv run dcindex stats                         # row counts + per-category coverage\nuv run dcindex events                        # list ingested DEF CON editions\nuv run dcindex search \"azure ad\"             # full-text search; prints a #id per result\nuv run dcindex search \"hyper\" --all          # every match (default shows 50; -n to set a limit)\nuv run dcindex show 213                       # full detail: speakers, abstract, and known URLs\n```\n\n**Search** matches **substrings** (a trigram index), so `hyper` finds `hypervisor`; terms shorter\nthan 3 characters fall back to a `LIKE` scan so `ai`/`os`/`5g` still work. The match summary is\nprinted to **stderr**, so stdout carries only result rows and pipes cleanly.\n\n**JSON output** for scripting — both `search` and `show` take `--json` and emit full session metadata\n(abstract, track, room, time, source URL, speakers with affiliation/bio, and all material links) on\nstdout, ready for `jq`:\n\n```bash\nuv run dcindex search \"hypervisor\" --json | jq '.[].title'    # array of full hit objects\nuv run dcindex search \"hypervisor\" --json --all | jq length   # every match\nuv run dcindex show 213 --json | jq '.materials'              # one full session object\n```\n\nEditions accept any of `dc30`, `defcon30`, `defcon-30`, `30`, or the year `2022`; when ingesting a\nfile the edition is also inferred from the filename (`dc30_mysqldump.txt`). Limit what you ingest\nwith `--categories` (e.g. `-c talk -c village`).\n\nData lives under `~/.local/share/dcindex/` (override with `--data-dir` or `DCINDEX_DATA_DIR`):\n`dcindex.sqlite3`, `dumps/` (cached dump files — download once, reuse forever), and `dcindex.log`.\n\n## Architecture\n\nLayered backend; the dependency rule points inward. The CLI (and a future TUI/API) call **only** the\nservice layer — never repositories, parsers, or adapters directly.\n\n```\ncli ─► services ─► parsers          (mysqldump reader, Outel schedule mapper, materials classifier)\n            │   └─► adapters         (DumpReader, HttpClient, DumpCache)\n            └─► storage              (sqlite3 + FTS5, repositories, snapshots/provenance)\n   everything ─► core / dto          (config, errors, editions, models; pydantic contracts)\n```\n\n- `ServiceContainer` is the DI root; `IngestService`, `SearchService`, `SessionService`,\n  `StatsService` are the public API.\n- Parsers are pure (SQL text → DTOs); all I/O (file reads, the one network fetch) lives in adapters.\n\n## Coverage\n\n| Years | Talks + speakers | Other activities | Materials |\n|---|---|---|---|\n| DC26–DC28 (2018–2020) | ✅ | — | links in talk descriptions |\n| DC29 (2021) | ✅ | villages | ✅ links |\n| DC30–DC31 (2022–2023) | ✅ | villages, demolabs, workshops, training, contests | ✅ links |\n| DC32–DC33 (2024–2025) | ✅ | villages, `pages` (consolidated activities) | ✅ links |\n\n`vendors` and CMS/utility tables (`articles`, `documents`, `map_matrix`, …) are intentionally skipped.\n\n## Development\n\n```bash\nuv run pytest        # unit + integration tests, fully offline against a synthetic fixture dump\nuv run ruff check .  # lint\n```\n\nThe integration suite pins the Phase-1 invariant that ingesting writes only the database + provenance\nrows — never a media/binary asset file.\n\n## License\n\n[MIT](LICENSE) © 2026 Evan H. Dygert.\n\ndcindex indexes only **metadata** (titles, abstracts, speakers, and links). It does not redistribute\nDEF CON or Outel content; all material links point back to their original sources. Respect the source\nsites' terms of use.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fedygert%2Fdcindex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fedygert%2Fdcindex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fedygert%2Fdcindex/lists"}