{"id":46639319,"url":"https://github.com/ericchansen/knowledge-enrichment-demo","last_synced_at":"2026-03-08T02:19:03.013Z","repository":{"id":339257750,"uuid":"1161141988","full_name":"ericchansen/knowledge-enrichment-demo","owner":"ericchansen","description":"Azure AI Content Understanding demo — overnight knowledge base enrichment using idle PTU capacity","archived":false,"fork":false,"pushed_at":"2026-02-18T19:50:00.000Z","size":80,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-18T23:43:13.589Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ericchansen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-18T19:28:07.000Z","updated_at":"2026-02-18T19:49:54.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ericchansen/knowledge-enrichment-demo","commit_stats":null,"previous_names":["ericchansen/knowledge-enrichment-demo"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/ericchansen/knowledge-enrichment-demo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ericchansen%2Fknowledge-enrichment-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ericchansen%2Fknowledge-enrichment-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ericchansen%2Fknowledge-enrichment-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ericchansen%2Fknowledge-enrichment-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ericchansen","download_url":"https://codeload.github.com/ericchansen/knowledge-enrichment-demo/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ericchansen%2Fknowledge-enrichment-demo/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30242406,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-08T00:58:18.660Z","status":"online","status_checked_at":"2026-03-08T02:00:06.215Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-03-08T02:19:02.172Z","updated_at":"2026-03-08T02:19:02.986Z","avatar_url":"https://github.com/ericchansen.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Knowledge Base Overnight Enrichment Demo\n\n**Demo: Azure AI Content Understanding for knowledge base enrichment**\n\nEnterprises with reserved PTU capacity often have idle compute during off-hours. This demo shows how Azure AI Content Understanding can enrich document knowledge bases overnight — extracting metadata, structure, and semantic fields — resulting in dramatically better RAG answers.\n\n## The Demo Story\n\n1. **Upload** a corpus of GAO cybersecurity reports (20 PDFs)\n2. **Baseline pipeline** — extract text, chunk, embed, index (standard RAG)\n3. **Enhanced pipeline** — use CU custom analyzers to extract report metadata (titles, agencies, categories, findings), then chunk, embed, and index with that metadata\n4. **Compare** — ask the same questions to both indexes side-by-side and see the quality difference\n\n## Quick Start\n\n### Prerequisites\n\n- Python 3.12 (pinned in `.python-version`)\n- [uv](https://docs.astral.sh/uv/) package manager\n- Azure subscription with: AI Content Understanding, AI Search, Azure OpenAI, Blob Storage\n\n### Setup\n\n```bash\n# Install dependencies\nuv sync\n\n# Configure environment\ncp .env.template .env\n# Edit .env with your Azure resource endpoints and keys\n\n# Run tests\nuv run pytest -v\n\n# Start the server\nuv run uvicorn enrichment.server:app --reload\n```\n\nOpen http://localhost:8080 for the side-by-side comparison UI.\n\n### Run the Enrichment Pipelines\n\n```bash\n# Download the GAO corpus (requires browser — see scripts/download_corpus.py)\n# Then run both pipelines:\nuv run python scripts/run_pipelines.py --corpus-dir data/corpus\n\n# Or run one at a time:\nuv run python scripts/run_pipelines.py --pipeline baseline\nuv run python scripts/run_pipelines.py --pipeline enhanced\n```\n\n### Docker\n\n```bash\ndocker build -t enrichment .\ndocker run -p 8080:8080 --env-file .env enrichment\n```\n\n## Architecture\n\n```\n┌──────────────┐     ┌───────────────────┐     ┌──────────────────┐\n│   PDF Corpus │────▶│  Content          │────▶│  Azure AI Search │\n│  (Blob Store)│     │  Understanding    │     │  (2 indexes)     │\n└──────────────┘     └───────────────────┘     └──────────────────┘\n                           │                          │\n                     ┌─────┴─────┐              ┌─────┴─────┐\n                     │ Baseline  │              │ Enhanced  │\n                     │ (text     │              │ (metadata │\n                     │  only)    │              │  + text)  │\n                     └───────────┘              └───────────┘\n                           │                          │\n                     ┌─────┴──────────────────────────┴─────┐\n                     │      Side-by-Side Comparison UI       │\n                     │  ┌─────────────┐  ┌────────────────┐  │\n                     │  │  Baseline   │  │  CU-Enhanced   │  │\n                     │  │    RAG      │  │     RAG        │  │\n                     │  └─────────────┘  └────────────────┘  │\n                     └──────────────────────────────────────┘\n```\n\n### Pipeline Details\n\n**Baseline**: CU prebuilt-layout → paragraph chunking → text-embedding-3-small → AI Search (text + vector)\n\n**Enhanced**: CU custom analyzer → same chunking → same embeddings → AI Search (text + vector + report_title, report_number, topic_category, executive_summary, agencies, section_title)\n\nThe enhanced index lets RAG answers cite specific reports, filter by agency or topic, and use executive summaries for better context.\n\n## Project Structure\n\n```text\n├── src/enrichment/\n│   ├── server.py                 # FastAPI app (API + static UI)\n│   ├── config.py                 # Pydantic settings from .env\n│   ├── models/                   # Pydantic data models\n│   ├── services/\n│   │   ├── content_understanding.py  # CU SDK integration\n│   │   ├── search.py                 # AI Search index management\n│   │   ├── embedding.py              # Azure OpenAI embeddings\n│   │   ├── chunking.py               # Paragraph-aware text splitter\n│   │   ├── chat.py                   # RAG chat (baseline + enhanced)\n│   │   └── storage.py                # Blob storage operations\n│   ├── pipeline/\n│   │   ├── baseline.py               # Baseline enrichment pipeline\n│   │   └── enhanced.py               # CU-enhanced pipeline\n│   └── static/                       # Comparison UI (HTML/CSS/JS)\n├── scripts/\n│   ├── download_corpus.py            # GAO report downloader\n│   └── run_pipelines.py              # Pipeline runner\n├── tests/                            # 86 tests, 85% coverage\n├── data/corpus/                      # Downloaded PDFs (gitignored)\n├── Dockerfile\n├── pyproject.toml\n└── .env.template\n```\n\n## Development\n\n```bash\nuv run pytest -v              # Run tests (80% coverage minimum)\nuv run ruff check .           # Lint\nuv run ruff format .          # Format\nuv run pyright                # Type check\nuv run pre-commit run --all-files  # All checks\n```\n\n## Tech Stack\n\n| Component | Technology |\n|-----------|------------|\n| **AI Extraction** | Azure AI Content Understanding |\n| **Search** | Azure AI Search (hybrid: keyword + vector) |\n| **Embeddings** | Azure OpenAI text-embedding-3-small |\n| **Chat** | Azure OpenAI GPT-4o |\n| **Runtime** | Python 3.12, FastAPI, uvicorn |\n| **Package Manager** | uv |\n| **Testing** | pytest (85% coverage) |\n| **Quality** | Ruff, Pyright, Snyk, detect-secrets |\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fericchansen%2Fknowledge-enrichment-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fericchansen%2Fknowledge-enrichment-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fericchansen%2Fknowledge-enrichment-demo/lists"}