{"id":50049578,"url":"https://github.com/cbeaulieu-gt/hackathon-llm-support-agent","last_synced_at":"2026-05-21T08:40:46.112Z","repository":{"id":358265673,"uuid":"1228443241","full_name":"cbeaulieu-gt/hackathon-llm-support-agent","owner":"cbeaulieu-gt","description":"Multi-stage support-ticket triage agent: BM25 retrieval, deterministic safety gates, grounded LLM responses across HackerRank/Claude/Visa support corpora. Built for the May 2026 HackerRank Orchestrate hackathon.","archived":false,"fork":false,"pushed_at":"2026-05-16T14:06:37.000Z","size":2511,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-16T15:48:03.022Z","etag":null,"topics":["ai-agents","anthropic","bm25","claude","customer-support","hackathon","llm","python","rag","triage"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cbeaulieu-gt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":"support_tickets/output.csv","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-04T02:48:04.000Z","updated_at":"2026-05-16T13:50:44.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/cbeaulieu-gt/hackathon-llm-support-agent","commit_stats":null,"previous_names":["cbeaulieu-gt/hackathon-llm-support-agent"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/cbeaulieu-gt/hackathon-llm-support-agent","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cbeaulieu-gt%2Fhackathon-llm-support-agent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cbeaulieu-gt%2Fhackathon-llm-support-agent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cbeaulieu-gt%2Fhackathon-llm-support-agent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cbeaulieu-gt%2Fhackathon-llm-support-agent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cbeaulieu-gt","download_url":"https://codeload.github.com/cbeaulieu-gt/hackathon-llm-support-agent/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cbeaulieu-gt%2Fhackathon-llm-support-agent/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33294031,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-21T02:57:32.698Z","status":"ssl_error","status_checked_at":"2026-05-21T02:57:31.990Z","response_time":62,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agents","anthropic","bm25","claude","customer-support","hackathon","llm","python","rag","triage"],"created_at":"2026-05-21T08:40:44.343Z","updated_at":"2026-05-21T08:40:46.106Z","avatar_url":"https://github.com/cbeaulieu-gt.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# hackathon-llm-support-agent\n\nMulti-stage support-ticket triage agent built for the May 2026 HackerRank Orchestrate hackathon.\n\n⭐ **Result: 65th place out of 12,885 participants (1,349 submissions)**\n\n## Context\n\nThis was my submission for the 24-hour HackerRank Orchestrate challenge (May 1–2, 2026). The problem: build an agent that triages support tickets across three product corpora — HackerRank, Claude, and Visa — using only the provided support documentation, no external knowledge. The repo here is the work I built; the hackathon scaffold has been removed.\n\nFull problem spec: [`docs/problem_statement.md`](./docs/problem_statement.md).\n\n## What it does\n\nReads `support_tickets/support_tickets.csv` (29 rows) and writes `support_tickets/output.csv` with five columns per row:\n\n| Column | Values |\n|---|---|\n| `status` | `replied`, `escalated` |\n| `product_area` | most relevant support category / domain area |\n| `response` | user-facing answer grounded in the corpus |\n| `justification` | concise explanation of the routing/answering decision |\n| `request_type` | `product_issue`, `feature_request`, `bug`, `invalid` |\n\n## Architecture\n\nEight stages: preprocessing (deterministic), safety triage (deterministic regex + keywords), domain routing (1 LLM call, only when `Company=None`), request-type classification (1 LLM call), BM25 retrieval (deterministic), product-area assignment (deterministic alias map), abstain gate (deterministic, 8-rule precedence), and response generation (≤1 LLM call with grounded JSON output).\n\nLLM calls are limited to stages 2, 3, and 7 — everything else is byte-stable across runs. Per-row LLM budget is 5 attempts shared across those three stages.\n\nFull stage table, test instructions, and output schema: [`code/README.md`](./code/README.md).\n\n## Engineering decisions worth calling out\n\n**Determinism first.** Stages 0/1/4/5/6 are byte-stable (sorted file iteration, regex normalization, deterministic alias map). Stages 2/3/7 run at `temperature=0`. Cost is ~$0.20 per full 29-row run on Sonnet 4.5; worst-case wall clock is ~3 minutes.\n\n**Reproducibility over cosmetic correction.** 28 of 29 rows are produced byte-for-byte by `python -m code.main`. Row 29 ships with a documented hallucination: the Stage 7 LLM confidently asserts that a US Virgin Islands merchant requiring a $10 minimum \"violates Visa's rules\" — the cited doc (line 202 of `data/visa/support.md`) explicitly names US Virgin Islands as an exception territory where the $10 minimum is permitted. I tried a jurisdictional-sensitivity prompt addition; it prevented the hallucination on row 29 but caused row 5 to escalate defensively. Net regressive. I kept the reproducible-but-wrong output and disclosed it rather than ship a prompt fix that broke other rows. Full disclosure and scoring trade-off analysis: [`docs/verification_findings.md`](./docs/verification_findings.md).\n\n**Explicit abstain gate with precedence order.** The 8-rule gate fires in fixed order: injection → high_risk → outage → action_impossible → billing → routing-failed → request_type-failed → no-retrieval. Escalation is the architecturally-correct fallback when grounding fails — the gate makes that explicit rather than letting the response-generation stage decide.\n\n**Tests written before code.** 145 unit tests across all 8 stages plus 7 end-to-end propagation tests on a fixture corpus. Tests were written per the TDD workflow documented in `docs/PLAN.md` before the stages were implemented. See [`code/README.md`](./code/README.md) for the test invocation.\n\n## Results\n\n12 escalated / 17 replied on the 29-row test set. Of the 17 replied rows: 14 verified grounded against the cited corpus document, 3 are templated out-of-scope responses (request type `invalid`), 1 known hallucination (row 29, disclosed above).\n\n## Run it\n\n```bash\npython -m venv .venv\n./.venv/Scripts/python.exe -m pip install -r code/requirements.txt\ncp .env.example .env   # add ANTHROPIC_API_KEY\n./.venv/Scripts/python.exe -m code.main\n```\n\nSee [`code/README.md`](./code/README.md) for the full invocation, sample validation, and test commands.\n\n## Repo layout\n\n```\n.\n├── code/              # Agent implementation (8-stage pipeline)\n├── data/              # Support corpora (HackerRank, Claude, Visa)\n├── docs/              # Design rationale, plan, verification findings\n└── support_tickets/   # Input CSV, sample CSV, output CSV\n```\n\n## Further reading\n\n- [`code/README.md`](./code/README.md) — architecture table, run commands, output schema, test invocation, cost\n- [`docs/PLAN.md`](./docs/PLAN.md) — 1598-line design rationale (Rev 3/4/5/5.1): per-stage failure modes, regex/keyword sets, testing methodology, adversarial review rounds\n- [`docs/verification_findings.md`](./docs/verification_findings.md) — pre-submission verification pass: hallucination disclosure, reproducibility refit, scoring trade-offs\n- [`docs/problem_statement.md`](./docs/problem_statement.md) — full problem spec, input/output schema, allowed values\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcbeaulieu-gt%2Fhackathon-llm-support-agent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcbeaulieu-gt%2Fhackathon-llm-support-agent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcbeaulieu-gt%2Fhackathon-llm-support-agent/lists"}