Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/itprodirect/exai-insurance-intel

AI-powered insurance intelligence toolkit for carrier research, expert lookup, contractor vetting, and CAT-loss investigation.
https://github.com/itprodirect/exai-insurance-intel

agent-search caching catastrophe cost-tracking evaluation exa insurance jupyter-notebook litigation-support people-search web-search

Last synced: 4 months ago
JSON representation

AI-powered insurance intelligence toolkit for carrier research, expert lookup, contractor vetting, and CAT-loss investigation.

Awesome Lists containing this project

README 

          
# exai-insurance-intel



Exa-powered **insurance intelligence toolkit** for CAT-loss, claims, expert, contractor, and market research workflows.



## What This Repo Is Today



This repo is a **backend workflow engine, API, and pilot web UI** for insurance intelligence. All core intelligence workflows (ranked search, cited answers, research reports, structured extraction, seed-URL discovery, benchmark evaluation) are fully implemented, tested, and runnable from the CLI, Jupyter notebook, FastAPI server, and Next.js frontend.



**What exists:**

- Complete CLI with 8 commands covering all Exa API endpoints

- FastAPI server wrapping all workflows as JSON endpoints

- Next.js frontend with search, answer, and research workflow UIs

- SQLite caching with budget enforcement and cost ledger

- Evaluation taxonomy with benchmark suites (55+ insurance queries)

- Artifact export system (JSON/JSONL/CSV/Markdown)

- Smoke + live execution modes with CI on every push

- Comprehensive test suite (~3,300 lines)



**What does not exist yet:**

- Container or cloud deployment (no Docker, no Terraform)

- Production database (SQLite cache only)

- Authentication or multi-user support



**Near-term direction:** Build a controlled pilot web product layer on top of the existing workflow engine. See [docs/roadmap.md](docs/roadmap.md) for phased tracks and [docs/pilot-architecture-decision.md](docs/pilot-architecture-decision.md) for architectural defaults.



This repo stays intentionally lean while supporting repeatable notebook and CLI workflows for insurance research, cited answers, structured extraction, and report generation.



Core local resources:

- one core notebook: `exa_people_search_eval.ipynb`

- one local env file: `.env` (not committed)

- one local sqlite cache: `exa_cache.sqlite` (not committed)



## Quick Navigation



- [What this repo evaluates](#what-this-repo-evaluates)

- [Feature matrix](#feature-matrix)

- [Architecture](#architecture)

- [Windows 11 setup](#windows-11-setup-python-310)

- [CLI commands](#cli-commands)

- [API server](#api-server)

- [Frontend](#frontend)

- [Experiment artifacts](#experiment-artifacts)

- [Demo gallery](#demo-gallery)

- [Integration boundaries](#integration-boundaries)

- [Manual Live Validation](#manual-live-validation)

- [Roadmap and delivery history](#roadmap-and-delivery-history)

- [Guardrails](#guardrails)



## Feature Matrix



| Capability | Status | Primary interface | Primary artifact/output | Notes |

| --- | --- | --- | --- | --- |

| Ranked search | Done | `python -m exa_demo search` | `results.jsonl`, `summary.json` | Supports additive deep-search controls |

| Benchmark evaluation | Done | `python -m exa_demo eval` | `queries.jsonl`, `results.jsonl`, `summary.json` | Includes taxonomy scoring and grouped comparison context |

| Search-type comparison | Done | `python -m exa_demo compare-search-types` | `comparison.md` plus paired run artifacts | Compares `deep` vs `deep-reasoning` end to end |

| Cited answers | Done | `python -m exa_demo answer` | `answer.json` | Separate workflow from ranked-search evaluation |

| Research reports | Done | `python -m exa_demo research` | `research.json` | Separate report-style workflow with citations |

| Structured extraction | Done | `python -m exa_demo structured-search` | `structured_output.json` | Uses `outputSchema` for schema-driven extraction |

| Seed-URL discovery | Done | `python -m exa_demo find-similar` | `find_similar.json` | Separate `/findSimilar` workflow and normalization path |

| Cache and budget ledger | Done | Notebook + CLI | `exa_cache.sqlite`, `summary.json` | Prevents re-billing on cache hits |

| API server | Done | `uvicorn exa_demo.api:app` | JSON responses | Thin FastAPI wrapper over CLI workflows; smoke mode first-class |

| Frontend UI | Done | `npm run dev` in `frontend/` | Browser UI | Next.js + TypeScript + Tailwind + shadcn/ui; search, answer, research |

| Smoke CI and local tests | Done | GitHub Actions + `pytest -q` | CI runs and local test suite | Default workflow runs `pytest` and notebook smoke |



## Architecture



```mermaid

flowchart TD

    U["Notebook or CLI user flow"] --> CLI["CLI / notebook orchestration"]

    CLI --> CFG["Config + runtime state"]

    CLI --> CACHE["SQLite cache + budget ledger"]

    CLI --> CLIENT["Exa client adapters"]

    CLIENT --> EXA["Exa endpoints\n/search, /answer, /research, /findSimilar"]

    CLIENT --> MOCK["Smoke-mode mocked responses"]

    CLIENT --> MODELS["Typed models + normalization"]

    MODELS --> EVAL["Evaluation + taxonomy + grouped comparison"]

    MODELS --> ART["Artifact writer"]

    EVAL --> ART

    ART --> EXP["experiments// artifacts"]

    EXP --> DOCS["README, roadmap, issue tracker, session notes"]

```



## Repo Map



- `src/exa_demo/cli.py`: thin CLI entrypoint; parser/runtime/compare helpers live in `cli_parser.py`, `cli_runtime.py`, and `cli_eval.py`

- `src/exa_demo/client.py`: stable transport facade; payload builders and smoke responders live in `client_payloads.py` and `client_smoke.py`

- `src/exa_demo/models.py`: stable model facade; shared API/result records live in `api_models.py`

- `src/exa_demo/reporting.py`: summary/research facade; comparison analysis/rendering lives in `comparison_reporting.py` and `comparison_analysis.py`

- `src/exa_demo/artifacts.py`: experiment writer; manifest/serialization helpers live in `artifact_manifest.py`

- `src/exa_demo/`: remaining reusable package modules for config, evaluation, workflows, cache, and safety

- `tests/`: CLI, client, model, artifact, script, and evaluation coverage

- `benchmarks/insurance_cat_queries.json`: named query suites used by notebook and CLI evaluation

- `experiments/`: runtime artifact root for workflow commands; local runs stay untracked by default

- `docs/roadmap.md`: canonical backlog and delivery phases

- `docs/issue-tracker.md`: GitHub issue-to-roadmap mapping

- `docs/sessions/`: durable session history



## What This Repo Evaluates



- Query relevance for CAT-loss / insurance professional discovery

- Cost per uncached query and projected spend at scale

- Repeatability via sqlite caching (reruns should not re-bill)

- Safe usage posture (public/professional info only)



## Recommended Cheap Defaults



In Cell 2 (`CONFIG`), start with:

- `use_highlights=True`

- `use_text=False`

- `use_summary=False`

- `num_results=5`



Optional for before/after reporting in notebook Cell 9:

- `CONFIG["compare_to_run_id"] = ""`

- `CONFIG["compare_base_dir"] = "experiments"`



## Windows 11 Setup (Python 3.10+)



1. Open PowerShell in this repo.

2. Create venv:



```powershell

py -3.10 -m venv .venv

```



3. Activate:



```powershell

.\.venv\Scripts\Activate.ps1

```



4. Install dependencies:



```powershell

python -m pip install --upgrade pip

pip install -r requirements.txt

```



## Installable Package and CLI



For an editable install with the CLI and dev tools available:



```powershell

pip install -e ".[dev]"

```



If your environment blocks build-dependency downloads, use:



```powershell

pip install -e ".[dev]" --no-build-isolation

```



Optional local hooks:



```powershell

pre-commit install

```

## Configure Environment



Create `.env` from template:



```powershell

Copy-Item .env.example .env

```



Edit `.env`:



```env

EXA_API_KEY=your_real_exa_api_key

EXA_SMOKE_NO_NETWORK=0

# Optional: set a run id label for grouped budget metrics

# EXA_RUN_ID=demo-2026-03-05

```



Notes:

- `.env` is ignored by git.

- If you want no-network testing, use smoke mode (`EXA_SMOKE_NO_NETWORK=1` or runner `--mode smoke`).



## Run Notebook



Open `exa_people_search_eval.ipynb` in Jupyter Lab or VS Code and run top-to-bottom.



Notebook flow is intentionally fixed to 9 cells:

1. Install/import + env load

2. Config

3. Exa call wrapper

4. Cache wrapper (sqlite) + budget enforcement

5. Single query demo

6. Batch query suite

7. Summary + qualitative notes

8. Cost projections

9. Decision rubric + integration recommendations



## CLI Commands



```powershell

python -m exa_demo search "forensic engineer insurance expert witness" --mode smoke --json

python -m exa_demo answer "What is the Florida appraisal clause dispute process?" --mode smoke --json

python -m exa_demo research "Summarize the Florida CAT market outlook." --mode smoke --json

python -m exa_demo structured-search "independent adjuster florida catastrophe claims" --schema-file .\path\to\structured-schema.json --mode smoke --json

python -m exa_demo find-similar "https://example.com/florida-appraisal-decision" --mode smoke --json

python -m exa_demo search "Florida property insurance appraisal clause" --type deep --additional-query "Florida appraisal dispute statute" --start-published-date 2025-01-01 --livecrawl --json

python -m exa_demo eval --mode smoke --suite forensic_and_damage_engineering --limit 5 --json

python -m exa_demo compare-search-types --mode smoke --suite forensic_and_damage_engineering --baseline-type deep --candidate-type deep-reasoning --limit 5 --json

python -m exa_demo eval --mode smoke --limit 5 --compare-to-run-id 20260310T033256Z --json

python -m exa_demo budget --run-id demo-2026-03 --json

```



The search and eval commands write the same `experiments//` artifact bundle as the notebook flow.

The `answer` command writes the same run directory and adds an `answer.json` artifact containing the cited-answer payload.

The `research` command writes the same run directory and adds `research.json` plus `research.md` artifacts for the research-report payload.

The `structured-search` command runs a schema-driven deep search and writes a `structured_output.json` artifact containing the extracted structured payload.

The `find-similar` command runs a seed-URL discovery workflow and writes a `find_similar.json` artifact containing the similar-result payload.

Eval workflows also emit additive export companions such as `results.csv`, `comparison.json`, `grouped_query_outcomes.csv`, and `manifest.json` without changing the existing JSON/JSONL contracts.

Deep-search-oriented request shaping is now exposed directly in the CLI with additive flags such as `--additional-query`, `--start-published-date`, `--end-published-date`, and `--livecrawl`.

Search cost estimation can also be overridden from the CLI for search-type experiments with flags such as `--deep-search-cost-1-25` and `--deep-reasoning-search-cost-1-25`.



Eval output now includes a taxonomy scorecard (relevance, credibility, actionability, confidence) and per-query failure reasons (`no_results`, `off_domain`, `low_confidence`).

Use `--compare-to-run-id` for before/after deltas across quality and failure rates when both runs share query text.

When comparison is enabled, the run also writes a human-readable `experiments//comparison.md` report with grouped query outcomes when suite context is available.

`compare-search-types` is the end-to-end workflow for running the same suite against two search types and emitting the grouped comparison bundle in one command.

`answer` is a separate cited-answer workflow and intentionally does not reuse the ranked-search evaluation taxonomy.

`structured-search` is a separate schema-driven extraction workflow and intentionally stores the raw structured payload outside the ranked-search `results.jsonl` path.

`find-similar` is a separate discovery workflow and intentionally stores the similar-result payload outside the ranked-search evaluation path.



## API Server



The API is a thin FastAPI wrapper over the same workflow functions the CLI uses. Install with the `api` extra:



```powershell

pip install -e ".[api]"

```



Start the server:



```powershell

uvicorn exa_demo.api:app --reload

```



The server runs on `http://127.0.0.1:8000` by default. Interactive docs are at `/docs`.



### Endpoints



| Method | Path | Request body | Description |

| --- | --- | --- | --- |

| `GET` | `/health` | — | Health check |

| `POST` | `/api/search` | `{"query": "...", "mode": "smoke"}` | Ranked search with evaluation |

| `POST` | `/api/answer` | `{"query": "...", "mode": "smoke"}` | Cited-answer workflow |

| `POST` | `/api/research` | `{"query": "...", "mode": "smoke"}` | Research report workflow |

| `POST` | `/api/find-similar` | `{"url": "...", "mode": "smoke"}` | Seed-URL discovery |

| `POST` | `/api/structured-search` | `{"query": "...", "output_schema": {...}, "mode": "smoke"}` | Schema-driven extraction |



All endpoints default to `smoke` mode. Set `"mode": "live"` for real Exa API calls (requires `EXA_API_KEY`).



### Quick smoke test



```powershell

curl -X POST http://127.0.0.1:8000/api/search -H "Content-Type: application/json" -d "{\"query\": \"forensic engineer insurance\", \"mode\": \"smoke\"}"

```



## Frontend



The frontend is a Next.js app in `frontend/` that calls the FastAPI backend through a server-side proxy (no CORS issues).



### Setup



```powershell

cd frontend

npm install

cp .env.local.example .env.local

```



### Local Development (backend + frontend)



Terminal 1 — start the API server:



```powershell

uvicorn exa_demo.api:app --reload

```



Terminal 2 — start the frontend:



```powershell

cd frontend

npm run dev

```



Open `http://localhost:3000`. The frontend proxies API calls to the backend at `http://127.0.0.1:8000` (configurable via `BACKEND_URL` in `frontend/.env.local`).



### What the frontend provides



- Health indicator showing backend connection status

- Tab-based workflow selector (Search, Answer, Research)

- Search: query input, result count, taxonomy scores, result cards with highlights

- Answer: question input, cited answer display with source links

- Research: topic input, report display with sources

- Loading spinners and error handling on all workflows



All workflows default to smoke mode — no API key or network needed for frontend development.



## Benchmark Fixture



The packaged regression-style query fixture lives at `benchmarks/insurance_cat_queries.json`.

The fixture now supports named suites while preserving the aggregate default set, and suites can be authored as plain query arrays or richer objects with suite metadata plus mixed string/object query entries.

Use `--suite all` for the full benchmark or pick a named segment such as `forensic_and_damage_engineering`, `restoration_and_mitigation`, `carrier_tpa_and_vendor_ecosystem`, or `regulatory_legislative_and_market_news`.

The notebook still owns execution and presentation, but Cell 6 now loads this fixture so the query set is reusable in tests and future CLI flows.



## Experiment Artifacts



Each notebook run now writes a versioned artifact bundle under `experiments//`:



- `config.json`

- `queries.jsonl`

- `results.jsonl`

- `summary.json`



Workflow-specific commands may also add:



- `answer.json`

- `research.json`

- `research.md`

- `find_similar.json`

- `structured_output.json`

- `results.csv`

- `comparison.json`

- `grouped_query_outcomes.csv`

- `manifest.json`



Smoke-mode runs keep the same artifact shape, but with mocked results and zero spend.

Every run also records runtime execution metadata in `config.json`, `summary.json`, and `manifest.json` so downstream review can distinguish smoke artifacts from live runs.

Local runtime outputs under `experiments/` are intentionally gitignored by default; the repo may still keep a small curated sample artifact set when needed for documentation or regression context.



## Demo Gallery



Use [docs/demo-gallery.md](docs/demo-gallery.md) as the top-level walkthrough for the shipped workflows:



- `search` for ranked discovery and evaluation

- `answer` for cited lookup questions

- `research` for report-style market or regulatory summaries

- `structured-search` for schema-driven extraction

- `find-similar` for seed-URL expansion

- `compare-search-types` for quality/cost tradeoff analysis



The gallery is deliberately command-first. It points at the exact entrypoint and artifact for each workflow without changing the project core.



## Integration Boundaries



Use [docs/integration-boundaries.md](docs/integration-boundaries.md) for the current smoke-vs-live execution contract, artifact expectations, and delivery rules.



## Manual Live Validation



Use the manual GitHub Actions workflow in [.github/workflows/live-validation.yml](.github/workflows/live-validation.yml) when you want a deliberate real-API validation pass against the shipped CLI workflows.



Local dry run:



```powershell

python .\scripts\run_live_validation.py --mode smoke

```



Guidance:

- default to `--mode smoke` locally unless you are intentionally validating live API behavior

- the manual workflow is bounded by design and uploads runtime artifacts for review

- `--include-comparison` is optional because it is materially more expensive than the default endpoint checks



## Cache + Budget Behavior



- Requests are cached in `exa_cache.sqlite` by payload hash.

- Repeated requests return cache hits and should not re-bill.

- Budget hard stop applies to **uncached calls in current `RUN_ID`**.

- Metrics still include all-time totals for visibility.



Reset cache safely:



```powershell

python .\scripts\reset_cache.py

```



Skip prompt:



```powershell

python .\scripts\reset_cache.py --yes

```



## Smoke Runner (nbclient)



```powershell

python .\scripts\run_notebook_smoke.py --mode smoke

```



## Local Quality Gate



- `python -m ruff check .`

- `python -m pytest -q`

- `pre-commit run --all-files`

- `python scripts/run_live_validation.py --mode smoke`



Modes:

- `--mode smoke`: forced no-network run (default)

- `--mode live`: real API calls (requires `EXA_API_KEY`)

- `--mode auto`: live if key exists, otherwise smoke



Recommended boundary:

- default to `--mode smoke` for development, CI, and docs validation

- use `--mode live` only for deliberate manual validation with a real key and human review



Optional timeout override:



```powershell

python .\scripts\run_notebook_smoke.py --mode smoke --timeout 180

```



## Safe Cost Tuning



In Cell 2 (`CONFIG`), adjust only these first:

- `num_results`

- `use_highlights`

- `use_text`

- `use_summary`



Guidance:

- Keep `num_results` low for first pass

- Keep text/summary off unless needed for second-pass review

- Estimator intentionally rejects unsupported high `num_results` ranges until pricing tiers are updated



## GitHub Sync



If needed:



```powershell

git init

git branch -M main

git add .

git commit -m "feat: minimal exa people search eval harness"

git remote add origin https://github.com/itprodirect/exai-insurance-intel.git

git push -u origin main

```



If remote already exists:



```powershell

git remote set-url origin https://github.com/itprodirect/exai-insurance-intel.git

git push -u origin main

```



## Deep-Dive Rebuild Review



For a from-scratch architecture critique and refactor roadmap, see `docs/rebuild_review.md`.



## Roadmap and Delivery History



- Canonical roadmap: [docs/roadmap.md](docs/roadmap.md)

- GitHub issue tracker mapping: [docs/issue-tracker.md](docs/issue-tracker.md)

- ADR index: [docs/adr/README.md](docs/adr/README.md)

- Session note template: [docs/sessions/README.md](docs/sessions/README.md)

- Latest implementation session: [docs/sessions/2026-03-20-phase4-refactor-decomposition.md](docs/sessions/2026-03-20-phase4-refactor-decomposition.md)



## Guardrails



- Public/professional info only

- No address hunting / doxxing

- No contact harvesting

- Redaction stays enabled in notebook output

- Human review required before operational use