{"id":49117982,"url":"https://github.com/langchain-samples/risk-assessment-agent","last_synced_at":"2026-04-21T08:43:11.479Z","repository":{"id":351677408,"uuid":"1203286556","full_name":"langchain-samples/risk-assessment-agent","owner":"langchain-samples","description":"  requirements for AI initiatives against the EU AI Act, AICM, and AIEU   frameworks. Built with LangChain, traced to LangSmith, compatible with    LangGraph Studio. Supports Gemini, OpenAI, and Vertex AI.    ","archived":false,"fork":false,"pushed_at":"2026-04-16T03:07:25.000Z","size":694,"stargazers_count":0,"open_issues_count":3,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-16T03:27:03.402Z","etag":null,"topics":["intermediate","langchain","langsmith","tracing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/langchain-samples.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-06T22:48:10.000Z","updated_at":"2026-04-16T01:37:11.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/langchain-samples/risk-assessment-agent","commit_stats":null,"previous_names":["langchain-samples/risk-assessment-agent"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/langchain-samples/risk-assessment-agent","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langchain-samples%2Frisk-assessment-agent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langchain-samples%2Frisk-assessment-agent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langchain-samples%2Frisk-assessment-agent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langchain-samples%2Frisk-assessment-agent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/langchain-samples","download_url":"https://codeload.github.com/langchain-samples/risk-assessment-agent/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langchain-samples%2Frisk-assessment-agent/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32084712,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-21T06:27:27.065Z","status":"ssl_error","status_checked_at":"2026-04-21T06:27:21.250Z","response_time":128,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["intermediate","langchain","langsmith","tracing"],"created_at":"2026-04-21T08:43:10.646Z","updated_at":"2026-04-21T08:43:11.467Z","avatar_url":"https://github.com/langchain-samples.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Risk Assessment \u0026 Governance Agent\n\nA conversational AI agent that helps organizations assess risk and compliance requirements for AI initiatives. Describe what you want to build, and the agent walks you through a structured assessment covering the **EU AI Act**, **AICM** (AI Compliance Management), and **AIEU** (AI Ethics in the EU).\n\n## What It Does\n\nThe agent acts as a GRC (Governance, Risk, Compliance) advisor. When you tell it about an AI system you're planning to build, it:\n\n1. **Asks clarifying questions** — understands your system's purpose, data, users, deployment context, and timeline\n2. **Dispatches subagents** to gather information:\n   - **Regulatory Research Agent** — searches the internet for current AI Act, AICM, and AIEU requirements relevant to your system\n   - **GRC Database Analyst Agent** — queries the organization's internal GRC database for existing controls, risks, audit findings, and compliance gaps\n   - **Risk Assessor Agent** — synthesizes everything into a structured risk assessment\n3. **Delivers a comprehensive assessment** including:\n   - AI Act risk classification (Unacceptable / High / Limited / Minimal)\n   - Specific regulatory obligations with article references\n   - Control gap analysis against existing organizational controls\n   - Prioritized recommendations (must-have / should-have / nice-to-have)\n\n## Architecture\n\n```\n┌──────────────────────────────────────────────┐\n│           Orchestrator Agent                 │\n│   (create_agent + system prompt)             │\n│                                              │\n│   Subagents called as tools:                 │\n│   ┌──────────────┐  ┌───────────────────┐    │\n│   │  Regulatory   │  │  GRC Database     │    │\n│   │  Research     │  │  Analyst          │    │\n│   │  (Tavily)     │  │  (SQLite)         │    │\n│   └──────────────┘  └───────────────────┘    │\n│   ┌──────────────────────────────────────┐   │\n│   │  Risk Assessor                       │   │\n│   │  (synthesizes findings)              │   │\n│   └──────────────────────────────────────┘   │\n└──────────────────────────────────────────────┘\n         │                        │\n    LangSmith               LangGraph Studio\n    (tracing)               (langgraph.json)\n```\n\n| Component | Demo | Production Swap |\n|-----------|------|-----------------|\n| LLM | Gemini (API key), OpenAI, or Vertex AI | Azure OpenAI, Bedrock, etc. |\n| Database | SQLite (`risk_governance.db`) | BigQuery, Snowflake, or any SQLAlchemy DB |\n| Search | Tavily | Stardog knowledge graph, internal KB |\n| Tracing | LangSmith | LangSmith |\n\n## Prerequisites\n\n- Python 3.11+\n- [uv](https://docs.astral.sh/uv/) (`curl -LsSf https://astral.sh/uv/install.sh | sh`)\n- [LangGraph CLI](https://langchain-ai.github.io/langgraph/cloud/reference/cli/) (`pipx install langgraph-cli`)\n- API keys (see below)\n\n## Setup\n\n```bash\n# Clone / navigate to project\ncd risk-assessment-agent\n\n# Install dependencies\nuv sync\n\n# Configure environment variables\ncp .env.example .env\n# Edit .env with your API keys\n```\n\n### Required API Keys\n\n| Key | Service | Get it at |\n|-----|---------|-----------|\n| `TAVILY_API_KEY` | Tavily search | https://tavily.com |\n| `LANGSMITH_API_KEY` | LangSmith tracing | https://smith.langchain.com |\n\nPlus one of the LLM provider keys below.\n\n### Choosing an LLM Provider\n\nSet `MODEL_PROVIDER` in `.env` to one of: `gemini` (default), `openai`, or `vertex`.\n\n#### Option A: Gemini (API key)\n\nThe simplest option — uses Google's Gemini API directly.\n\n```bash\n# .env\nGOOGLE_API_KEY=your_key_here\nMODEL_PROVIDER=gemini\n```\n\nGet a key at https://aistudio.google.com/apikey\n\n#### Option B: OpenAI\n\n```bash\n# .env\nOPENAI_API_KEY=your_key_here\nMODEL_PROVIDER=openai\n```\n\nGet a key at https://platform.openai.com/api-keys\n\n#### Option C: Vertex AI\n\nUses Google Cloud's Vertex AI with a service account — the enterprise option for production deployments.\n\n1. **Create a GCP service account** with the `Vertex AI User` role\n2. **Download the service account JSON key** and save it as `vertexCred.json` in the project root\n3. **Configure `.env`**:\n\n```bash\n# .env\nGOOGLE_APPLICATION_CREDENTIALS=./vertexCred.json\nMODEL_PROVIDER=vertex\n```\n\n\u003e `vertexCred.json` is in `.gitignore` and will not be committed. Never check credentials into source control.\n\n### Seed the Database\n\n```bash\nuv run python seed_db.py\n```\n\nThis creates `risk_governance.db` with synthetic GRC data: 10 regulatory frameworks (including AI Act, AICM, AIEU), 12 policies, 35 controls, 22 risks, 28 mitigations, 18 audit findings, and 48 compliance mappings.\n\n## Running\n\n### Option 1: LangGraph Studio (recommended for demos)\n\n```bash\nlanggraph dev\n```\n\nThis starts the LangGraph development server and opens LangGraph Studio in your browser. The agent appears as **\"Risk Assessment Agent\"** in the Studio UI.\n\n- API: http://127.0.0.1:2024\n- Studio UI: opens automatically (or visit https://smith.langchain.com/studio/?baseUrl=http://127.0.0.1:2024)\n- API Docs: http://127.0.0.1:2024/docs\n\n\u003e **Note**: If `langgraph dev` can't find project dependencies, install them into the LangGraph CLI environment:\n\u003e ```bash\n\u003e pipx runpip langgraph-cli install -e .\n\u003e ```\n\n### Option 2: Interactive Chat (terminal)\n\n```bash\nuv run python chat.py\n```\n\nRich-formatted terminal chat with conversation memory. Type `quit` or `exit` to end.\n\n### Option 3: LangGraph API (programmatic)\n\n```bash\n# Start the server\nlanggraph dev --no-browser\n\n# Create a thread\ncurl -s http://127.0.0.1:2024/threads \\\n  -H \"Content-Type: application/json\" -d '{}'\n\n# Send a message (replace THREAD_ID)\ncurl -s \"http://127.0.0.1:2024/threads/THREAD_ID/runs/wait\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"assistant_id\": \"Risk Assessment Agent\",\n    \"input\": {\n      \"messages\": [{\"role\": \"user\", \"content\": \"I want to build an AI credit scoring system for EU lending decisions.\"}]\n    }\n  }'\n```\n\n## Sample Conversation\n\n```\nYou: I want to build an AI-powered customer credit scoring system that will be\n     used to make lending decisions for customers in the EU.\n\nAgent: That's a significant initiative! To give you the most accurate assessment,\n       could you tell me more about it?\n       - What specific types of data will the system use?\n       - Will it make fully automated decisions or assist loan officers?\n       - What is your target deployment timeline?\n\nYou: It uses historical loan data, income, and credit bureau data. It will assist\n     loan officers but also auto-reject below a threshold. Deploy in 6 months.\n\nAgent: [Dispatches regulatory research, GRC database analysis, and risk assessment subagents]\n\n       ## AI Credit Scoring System: Risk Assessment\n\n       ### AI Act Risk Classification: HIGH-RISK\n       Classified under EU AI Act Annex III, point 5(b) — AI systems used to\n       evaluate creditworthiness of natural persons...\n\n       ### Key Regulatory Obligations\n       - Art. 9: Risk Management System (lifecycle)\n       - Art. 10: Data Governance (bias detection)\n       - Art. 14: Human Oversight (override capability)\n       - Art. 43: Conformity Assessment (before deployment)\n       ...\n\n       ### Critical Gaps\n       - No conformity assessment process exists\n       - No AI system registration process for EU database\n       - Human override not available for auto-rejection decisions\n       ...\n```\n\n## Pre-populating LangSmith Traces\n\nTo generate realistic trace data for demos or evaluation, run the simulation script:\n\n```bash\nuv run python populate_traces.py\n```\n\nThis uses [openevals multi-turn simulation](https://docs.langchain.com/langsmith/multi-turn-simulation) to run 8 scenarios with simulated user personas (VP of Engineering, HR Director, CISO, etc.) that interact naturally with the agent over multiple turns. Each run generates slightly different conversations since the simulated users respond dynamically to the agent's questions.\n\n| Scenario | Persona | Turns | What it covers |\n|----------|---------|-------|----------------|\n| AI Credit Scoring System | VP of Engineering | 4 | High-risk financial AI |\n| AI Hiring Screening Tool | HR Director | 4 | High-risk employment AI |\n| Customer Service Chatbot | Product Manager | 4 | Limited-risk AI |\n| Fraud Detection System | Chief Risk Officer | 4 | High-risk with human-in-the-loop |\n| AI Medical Triage Assistant | Healthtech CTO | 5 | High-risk healthcare AI |\n| Compliance Status Review | Compliance Officer | 3 | GRC database queries |\n| AI Content Moderation | Trust \u0026 Safety Director | 4 | High-risk content moderation |\n| Open Risk Register Review | CISO | 3 | Risk register queries |\n\n\u003e **Note**: The simulated users require an OpenAI API key (`OPENAI_API_KEY` in `.env`) since they use `gpt-4o-mini`. The agent itself can use any configured provider.\n\n## Offline Evaluations\n\nThe project includes an evaluation suite that tests the agent across 4 models using a synthetic dataset uploaded to LangSmith.\n\n### 1. Create the Dataset\n\n```bash\nuv run python eval_dataset.py\n```\n\nThis creates a LangSmith dataset called **\"Risk Assessment Agent Evaluations\"** with 8 synthetic examples covering:\n\n| Example | Expected Risk Level | Key Tools |\n|---------|-------------------|-----------|\n| AI Credit Scoring System | High | All 3 subagents |\n| AI Hiring Screening Tool | High | All 3 subagents |\n| Customer Service Chatbot | Limited | All 3 subagents |\n| AI Fraud Detection System | High | All 3 subagents |\n| AI Medical Triage Assistant | High | All 3 subagents |\n| AI Content Moderation System | High | All 3 subagents |\n| Compliance Status Review | N/A | grc_database_analysis |\n| AI Predictive Maintenance | Minimal | All 3 subagents |\n\nEach example includes expected outputs: risk classification, expected tool calls, regulatory frameworks to cite, and assessment sections to include.\n\n### 2. Run Evaluations\n\n```bash\nuv run python offline_evals.py\n```\n\nThis runs 4 experiments — one per model — against the dataset. By default it uses:\n\n| Model | Provider |\n|-------|----------|\n| `gpt-4.1-mini` | OpenAI |\n| `gpt-4.1` | OpenAI |\n| `claude-sonnet-4-20250514` | Anthropic |\n| `gemini-2.5-flash` | Google |\n\n#### Customizing models\n\nBoth the agent models and the LLM judge model are configurable in `offline_evals.py`. You can use any model supported by LangChain's [`init_chat_model`](https://python.langchain.com/docs/how_to/chat_models_universal_init/).\n\n**Agent models** — edit the `MODEL_CONFIGS` list to add, remove, or swap models. Each entry runs a separate experiment:\n\n```python\n# offline_evals.py\nMODEL_CONFIGS = [\n    {\"model\": \"google_genai:gemini-2.5-flash\", \"temperature\": 0, \"label\": \"gemini-2.5-flash\"},\n    {\"model\": \"google_genai:gemini-2.5-pro\", \"temperature\": 0, \"label\": \"gemini-2.5-pro\"},\n    # {\"model\": \"openai:gpt-4.1\", \"temperature\": 0, \"label\": \"gpt-4.1\"},  # commented out = skipped\n]\n```\n\n**LLM judge model** — the two LLM judge evaluators (`subagent_delegation_quality`, `risk_classification_accuracy`) use a separate model to grade outputs. Change it by editing the `delegation_judge` and `classification_judge` lines:\n\n```python\n# offline_evals.py — change the judge model\ndelegation_judge = ChatOpenAI(model=\"gpt-4.1\", temperature=0).with_structured_output(...)\n# or use any LangChain chat model, e.g.:\n# from langchain_google_genai import ChatGoogleGenerativeAI\n# delegation_judge = ChatGoogleGenerativeAI(model=\"gemini-2.5-flash\", temperature=0).with_structured_output(...)\n```\n\nThe model string format is `provider:model_name` — for example `openai:gpt-4.1-mini`, `anthropic:claude-sonnet-4-20250514`, `google_genai:gemini-2.5-flash`. You only need the API key for the providers you're actually using.\n\n#### Evaluators\n\nEach experiment scores all 8 dataset examples with 4 evaluators:\n\n| Evaluator | Type | What It Measures |\n|-----------|------|-----------------|\n| `subagent_delegation_quality` | LLM Judge (trajectory) | Did the agent call the right subagents for the query? |\n| `risk_classification_accuracy` | LLM Judge (single step) | Is the EU AI Act risk classification correct? |\n| `regulatory_framework_coverage` | Custom Code | Are expected frameworks (AI Act, AICM, AIEU) cited? |\n| `assessment_structure_completeness` | Custom Code | Does the response include all required sections? |\n\nResults are visible in the LangSmith **Experiments** tab with model, prompt, and tool metadata populated for comparison.\n\n## LangSmith Tracing\n\nAll traces are automatically sent to LangSmith when `LANGSMITH_TRACING=true` is set. Traces include the full orchestrator + subagent hierarchy — you can see each subagent's tool calls (SQL queries, web searches) nested under the parent trace.\n\nView traces at: https://smith.langchain.com (under the `risk-assessment-agent` project)\n\n## Dependencies\n\n```\nlangchain\u003e=1.0,\u003c2.0\nlangchain-core\u003e=1.0,\u003c2.0\nlangchain-anthropic\u003e=0.3.0\nlangchain-google-genai\u003e=2.1.0\nlangchain-google-vertexai\u003e=3.2.0\nlangchain-openai\u003e=0.3.0\nlangchain-community\u003e=0.3.0,\u003c0.4.0\nlangchain-tavily\nlanggraph\u003e=1.0,\u003c2.0\nlangsmith\u003e=0.3.0\nsqlalchemy\u003e=2.0.0\npython-dotenv\u003e=1.0.0\nrich\u003e=13.0.0\nopenevals\u003e=0.1.0\n```\n\n## File Structure\n\n```\nrisk-assessment-agent/\n├── agent.py              # Orchestrator agent with subagent tools, exports `agent` for Studio\n├── tools.py              # Subagent definitions (regulatory, GRC analyst, risk assessor) + tool wrappers\n├── chat.py               # Interactive terminal chat loop (rich UI)\n├── seed_db.py            # Creates synthetic GRC SQLite database\n├── populate_traces.py    # Multi-turn simulation script to pre-populate LangSmith traces\n├── eval_dataset.py       # Synthetic evaluation dataset — creates \u0026 uploads to LangSmith\n├── offline_evals.py      # Offline evaluations — 4 evaluators × 4 models\n├── risk_governance.db    # SQLite database (generated by seed_db.py)\n├── langgraph.json        # LangGraph Studio / CLI configuration\n├── .env.example          # API key template\n├── .env                  # Your API keys (not committed)\n├── pyproject.toml        # Project dependencies\n└── README.md\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flangchain-samples%2Frisk-assessment-agent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flangchain-samples%2Frisk-assessment-agent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flangchain-samples%2Frisk-assessment-agent/lists"}