{"id":51593307,"url":"https://github.com/leestott/opustest","last_synced_at":"2026-07-11T16:30:53.716Z","repository":{"id":343225287,"uuid":"1176772751","full_name":"leestott/opustest","owner":"leestott","description":"opustest is an agentic AI system that automatically analyzes Python codebases and produces a detailed quality report. It uses multiple specialized AI agents — built with Microsoft Agent Framework that collaborate in a pipeline to score your code across four areas and flag every issue found.","archived":false,"fork":false,"pushed_at":"2026-03-09T16:32:51.000Z","size":74,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-09T16:56:36.703Z","etag":null,"topics":["agent","agentframework","foundry","microsoft"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/leestott.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-03-09T11:08:34.000Z","updated_at":"2026-03-09T16:50:16.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/leestott/opustest","commit_stats":null,"previous_names":["leestott/opustest"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/leestott/opustest","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leestott%2Fopustest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leestott%2Fopustest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leestott%2Fopustest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leestott%2Fopustest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/leestott","download_url":"https://codeload.github.com/leestott/opustest/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leestott%2Fopustest/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35368766,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-11T02:00:05.354Z","response_time":104,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent","agentframework","foundry","microsoft"],"created_at":"2026-07-11T16:30:52.937Z","updated_at":"2026-07-11T16:30:53.708Z","avatar_url":"https://github.com/leestott.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Opustest — AI-Powered Python Code Verification\n\n[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue?logo=python\u0026logoColor=white)](https://www.python.org/downloads/)\n[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)\n[![Microsoft Agent Framework](https://img.shields.io/badge/agents-Microsoft%20Agent%20Framework-purple?logo=microsoft)](https://github.com/microsoft/agent-framework)\n[![Azure Cosmos DB](https://img.shields.io/badge/database-Azure%20Cosmos%20DB-blue?logo=microsoftazure)](https://learn.microsoft.com/azure/cosmos-db/)\n[![Azure Container Apps](https://img.shields.io/badge/hosting-Azure%20Container%20Apps-blue?logo=microsoftazure)](https://learn.microsoft.com/azure/container-apps/)\n[![FastAPI](https://img.shields.io/badge/backend-FastAPI-009688?logo=fastapi\u0026logoColor=white)](https://fastapi.tiangolo.com/)\n\n---\n\n## What is opustest?\n\nopustest is an **agentic AI system** that automatically analyzes Python codebases and produces a detailed quality report. It uses multiple specialized AI agents — built with [Microsoft Agent Framework](https://github.com/microsoft/agent-framework) — that collaborate in a pipeline to score your code across four areas and flag every issue found.\n\nThe system uses **Retrieval-Augmented Generation (RAG)**: before analyzing your code, it retrieves curated examples of good and bad Python code from an **Azure Cosmos DB** database, and uses those examples as the quality standards for its review.\n\nYou interact with opustest through a **web-based UI** where you enter a **directory path** to a Python codebase, watch real-time progress updates, and receive an HTML report.\n\n### Who is this for?\n\n- **Developers** who want automated code reviews against consistent standards\n- **Teams** looking to enforce coding standards across projects\n- **Learners** exploring multi-agent AI systems, RAG patterns, and Azure deployment\n\n### What does the report look like?\n\nThe generated HTML report contains:\n\n1. **Scores (0–5)** in four areas, plus a total out of 20:\n   - Static Code Quality and Coding Standards\n   - Functional Correctness\n   - Handling of Known Errors\n   - Handling of Unknown Errors\n2. **Error table** listing every issue found, with columns:\n   - Error Found\n   - File\n   - Type of Error\n   - Explanation (why it was flagged, referencing the RAG database examples)\n   - Fix Prompt (a ready-to-use prompt for a coding assistant to fix the issue)\n\n---\n\n## Screenshots\n\n### Landing page — Git URL mode (default)\n![Landing page](screenshots/01-landing-page.png)\n\n### Local Path mode\n![Local Path mode](screenshots/02-local-path-mode.png)\n\n### Repository URL entered\n![URL entered](screenshots/03-url-entered.png)\n\n### Stage 1 — Code example retrieval (RAG)\n![Stage 1 retrieval](screenshots/04-stage1-retrieval.png)\n\n### Stage 3 — Verification agents running\n![Stage 3 verification](screenshots/05-stage3-verification.png)\n\n### Stage 4 — Report generation\n![Stage 4 report generation](screenshots/06-stage4-report-generation.png)\n\n### Verification complete\n![Verification complete](screenshots/07-verification-complete.png)\n\n### Report detail — scores and error table\n![Report detail](screenshots/08-report-detail.png)\n\n---\n\n## Architecture\n\nThe system uses these agents, orchestrated sequentially:\n\n1. **Code Example Retrieval Agent** — Retrieves Python code examples (good/bad) from Azure Cosmos DB via RAG\n2. **Codebase Import Agent** — Reads all `.py` files from the user-specified directory\n3. **Verification Agents** (one per scoring area):\n   - Code Quality and Coding Standards (score 0–5)\n   - Functional Correctness (score 0–5)\n   - Handling of Known Errors (score 0–5)\n   - Handling of Unknown Errors (score 0–5)\n4. **Report Generation Agent** — Produces an HTML report with scores and an error table\n\nAn **Orchestrator** coordinates the pipeline and streams progress updates to the web UI via SSE.\n\n### How it works (step by step)\n\n```mermaid\nflowchart TD\n    subgraph UI[\"🌐 Web UI (Browser)\"]\n        A[\"Enter directory path\\n\u0026 click Verify\"]\n    end\n\n    subgraph Server[\"⚡ FastAPI Backend\"]\n        B[\"POST /api/verify\\nSSE progress stream\"]\n    end\n\n    subgraph Orchestrator[\"🎯 Orchestrator Agent\"]\n        direction TB\n        C[\"Coordinates pipeline\\n\u0026 streams progress\"]\n    end\n\n    subgraph Stage1[\"📚 Stage 1: RAG Retrieval\"]\n        D[\"Code Example\\nRetrieval Agent\"]\n        E[(\"Azure Cosmos DB\\n(good/bad examples)\")]\n    end\n\n    subgraph Stage2[\"📂 Stage 2: Import\"]\n        F[\"Codebase\\nImport Agent\"]\n        G[(\"Local .py files\")]\n    end\n\n    subgraph Stage3[\"🔍 Stage 3: Verification\"]\n        H[\"Code Quality\\nAgent\"]\n        I[\"Functional\\nCorrectness Agent\"]\n        J[\"Known Errors\\nAgent\"]\n        K[\"Unknown Errors\\nAgent\"]\n    end\n\n    subgraph Stage4[\"📝 Stage 4: Report\"]\n        L[\"Report Generation\\nAgent\"]\n    end\n\n    M[\"📄 HTML Report\\n(scores + error table)\"]\n\n    A --\u003e|\"directory path\"| B\n    B --\u003e|\"start pipeline\"| C\n    C --\u003e D\n    D \u003c--\u003e|\"query Python examples\"| E\n    C --\u003e F\n    F \u003c--\u003e|\"read .py files\"| G\n    C --\u003e H \u0026 I \u0026 J \u0026 K\n    C --\u003e L\n    L --\u003e M\n    M --\u003e|\"SSE stream\"| B\n    B --\u003e|\"progress + report\"| A\n\n    style UI fill:#e3f2fd,stroke:#1565c0,color:#000\n    style Server fill:#fff3e0,stroke:#e65100,color:#000\n    style Orchestrator fill:#f3e5f5,stroke:#6a1b9a,color:#000\n    style Stage1 fill:#e8f5e9,stroke:#2e7d32,color:#000\n    style Stage2 fill:#fff8e1,stroke:#f9a825,color:#000\n    style Stage3 fill:#fce4ec,stroke:#c62828,color:#000\n    style Stage4 fill:#e0f2f1,stroke:#00695c,color:#000\n```\n\n1. You enter a **directory path** to a Python codebase in the **Web UI** and click **Verify Codebase**\n2. The **Orchestrator** starts the pipeline and streams progress back to the browser via Server-Sent Events (SSE)\n3. The **Code Example Retrieval Agent** queries Cosmos DB for Python examples (good and bad)\n4. The **Codebase Import Agent** reads all `.py` files from the directory\n5. Four **Verification Agents** each score one area (0–5) and list issues found\n6. The **Report Generation Agent** compiles everything into an HTML report\n7. The report is displayed in the Web UI\n\n### Project structure\n\n```\nopustest/\n├── backend/\n│   ├── agents/\n│   │   ├── code_example_retrieval.py   # RAG retrieval from Cosmos DB\n│   │   ├── codebase_import.py          # Reads .py files from directory\n│   │   ├── orchestrator.py             # Coordinates the full pipeline\n│   │   ├── report_generation.py        # Generates the HTML report\n│   │   └── verification/\n│   │       ├── code_quality.py         # Score: code quality \u0026 standards\n│   │       ├── functional_correctness.py # Score: functional correctness\n│   │       ├── known_errors.py         # Score: known error handling\n│   │       └── unknown_errors.py       # Score: unknown error handling\n│   ├── app.py                          # FastAPI server with SSE\n│   ├── config.py                       # Environment variable loading\n│   ├── cosmos_client.py                # Cosmos DB query functions\n│   └── git_utils.py                    # Git repo cloning for cloud mode\n├── frontend/\n│   ├── index.html                      # Web UI\n│   ├── styles.css                      # Styling\n│   └── app.js                          # SSE client \u0026 progress display\n├── infra/\n│   ├── main.bicep                      # Azure infra entry point\n│   ├── main.parameters.json            # azd parameter bindings\n│   └── modules/\n│       ├── acr.bicep                   # Azure Container Registry\n│       ├── container-app.bicep         # Container Apps Environment + App\n│       └── cosmos.bicep                # Cosmos DB account, database, container\n├── scripts/\n│   ├── deploy.ps1                      # One-command deploy (PowerShell)\n│   ├── deploy.sh                       # One-command deploy (Bash)\n│   ├── postprovision.ps1               # azd hook: seeds Cosmos DB after provision (Windows)\n│   ├── postprovision.sh                # azd hook: seeds Cosmos DB after provision (Linux/macOS)\n│   └── seed_cosmos.py                  # Populate sample code examples\n├── azure.yaml                          # azd project definition\n├── Dockerfile                          # Container image definition\n├── requirements.txt                    # Python dependencies\n└── .env.example                        # Template for environment variables\n```\n\n---\n\n## Cosmos DB Schema\n\nEach document in the code examples container has:\n\n| Field         | Description                                      |\n|---------------|--------------------------------------------------|\n| `type`        | `\"good\"` or `\"bad\"`                              |\n| `language`    | Programming language (only Python examples used)  |\n| `severity`    | `\"low\"`, `\"medium\"`, or `\"high\"`                 |\n| `description` | Explanation of what is good or bad                |\n| `code`        | The example code snippet                          |\n\n---\n\n## Getting Started\n\nThere are two ways to run opustest: **locally** (for development) or **deployed to Azure Container Apps** (for production). Both paths are covered below.\n\n### Prerequisites\n\nYou will need:\n\n| Tool | Purpose | Install link |\n|------|---------|-------------|\n| **Python 3.10+** | Run the backend and seed script | [python.org](https://www.python.org/downloads/) |\n| **Azure CLI** (`az`) | Authenticate and manage Azure resources | [Install Azure CLI](https://learn.microsoft.com/cli/azure/install-azure-cli) |\n| **Azure Developer CLI** (`azd`) | Provision infrastructure from Bicep | [Install azd](https://learn.microsoft.com/azure/developer/azure-developer-cli/install-azd) |\n| **Docker** | Build container images (cloud deploy only) | [Get Docker](https://docs.docker.com/get-docker/) |\n\nYou will also need:\n- An **Azure OpenAI** deployment with the Responses API enabled\n- An **Azure subscription** (for Cosmos DB and Container Apps)\n\n---\n\n### Option A: Run Locally\n\nFollow these steps to run opustest on your machine.\n\n**Step 1 — Clone and configure environment variables**\n\n```bash\ngit clone \u003cyour-repo-url\u003e\ncd opustest\ncp .env.example .env\n```\n\nOpen `.env` in a text editor and fill in your values:\n\n```\nAZURE_AI_PROJECT_ENDPOINT=https://\u003cyour-project\u003e.openai.azure.com/\nAZURE_AI_MODEL_DEPLOYMENT_NAME=gpt-4o\nCOSMOS_ENDPOINT=https://\u003cyour-account\u003e.documents.azure.com:443/\nCOSMOS_KEY=\u003cyour-cosmos-key\u003e\nCOSMOS_DATABASE_NAME=code-examples\nCOSMOS_CONTAINER_NAME=examples\n```\n\n\u003e **Where do these values come from?** `AZURE_AI_PROJECT_ENDPOINT` and `AZURE_AI_MODEL_DEPLOYMENT_NAME` come from your Azure OpenAI resource. The Cosmos values come from provisioning (Step 3) or from an existing Cosmos DB account.\n\n**Step 2 — Install Python dependencies**\n\n```bash\npip install --pre -r requirements.txt\n```\n\n\u003e The `--pre` flag is required because Microsoft Agent Framework is currently in preview.\n\n**Step 3 — Provision Cosmos DB and seed sample data**\n\n```bash\n# Log in to Azure\naz login\nazd auth login\n\n# Provision the Cosmos DB account, database, and container\nazd provision\n\n# Retrieve the Cosmos DB key and add it to your .env\naz cosmosdb keys list \\\n    --name \u003cCOSMOS_ACCOUNT_NAME\u003e \\\n    --resource-group \u003cAZURE_RESOURCE_GROUP\u003e \\\n    --query primaryMasterKey -o tsv\n```\n\nAfter provisioning, `azd` makes these outputs available:\n\n| Output                 | Description                        |\n|------------------------|------------------------------------|\n| `COSMOS_ENDPOINT`      | Cosmos DB account endpoint URL     |\n| `COSMOS_ACCOUNT_NAME`  | Cosmos DB account name             |\n| `COSMOS_DATABASE_NAME` | Database name (`code-examples`)    |\n| `COSMOS_CONTAINER_NAME`| Container name (`examples`)        |\n\nNow populate the database with sample good/bad Python code examples:\n\n```bash\npython scripts/seed_cosmos.py\n```\n\nThe script creates 19 sample documents:\n- **Good Python examples** (8): PEP 8 naming, type hints, specific exception handling, context managers, input validation, defensive programming with logging\n- **Bad Python examples** (9): bare except, poor naming, missing error handling, SQL injection, mutable defaults, swallowed exceptions, inconsistent return types\n- **Non-Python examples** (2): one JavaScript and one Java entry that the RAG filter correctly ignores\n\n\u003e **Tip:** When you deploy to Azure with `azd up`, the seed script runs automatically via the `postprovision` hook — you don't need to run it manually.\n\n**Step 4 — Start the server**\n\n```bash\nuvicorn backend.app:app --reload\n```\n\n**Step 5 — Use the app**\n\nOpen [http://localhost:8000](http://localhost:8000) in your browser.\n\nEnter the absolute path to a Python codebase directory (e.g. `C:\\Users\\you\\my-project` or `/home/you/my-project`) and click **Verify Codebase**.\n\nYou will see real-time progress updates for each stage, and the final HTML report will be displayed when complete.\n\n---\n\n### Option B: Deploy to Azure Container Apps\n\nThe recommended way to deploy is with `azd up`, which provisions all Azure resources, builds the Docker image, pushes it to ACR, deploys to Container Apps, and **automatically seeds Cosmos DB** with sample data via the `postprovision` hook.\n\n#### Quick deploy with azd\n\n```bash\n# Log in to Azure\naz login\nazd auth login\n\n# Set required environment variables\nazd init\nazd env set AZURE_AI_PROJECT_ENDPOINT \"https://\u003cyour-project\u003e.openai.azure.com/\"\nazd env set AZURE_AI_MODEL_DEPLOYMENT_NAME \"gpt-4o\"\n\n# Provision, build, and deploy (seeds Cosmos DB automatically)\nazd up\n```\n\nAfter deployment completes, `azd` prints the application URL.\n\nAlternatively, you can use the one-command deploy scripts:\n\n#### One-command deploy (PowerShell)\n\n```powershell\n.\\scripts\\deploy.ps1 `\n    -EnvironmentName codeverify `\n    -Location eastus `\n    -AzureAiProjectEndpoint \"https://\u003cyour-project\u003e.openai.azure.com/\"\n```\n\n#### One-command deploy (Bash)\n\n```bash\nchmod +x scripts/deploy.sh\n./scripts/deploy.sh \\\n    --env-name codeverify \\\n    --location eastus \\\n    --ai-endpoint \"https://\u003cyour-project\u003e.openai.azure.com/\"\n```\n\n#### What the script does\n\n1. Initialises an `azd` environment with your settings\n2. Provisions all Azure resources via Bicep (Cosmos DB, ACR, Container Apps)\n3. Builds the Docker image and pushes it to ACR\n4. Updates the Container App with the new image\n5. Seeds Cosmos DB with sample code examples (pass `-SkipSeed` / `--skip-seed` to skip)\n6. Prints the application URL\n\n#### Redeploying after code changes\n\n```bash\ndocker build -t code-verification:latest .\ndocker tag code-verification:latest \u003cACR_LOGIN_SERVER\u003e/code-verification:latest\ndocker push \u003cACR_LOGIN_SERVER\u003e/code-verification:latest\naz containerapp update --name \u003cAPP_NAME\u003e --resource-group \u003cRG_NAME\u003e --image \u003cACR_LOGIN_SERVER\u003e/code-verification:latest\n```\n\n#### Using the deployed app\n\nOnce deployed, open the application URL printed by `azd` (e.g. `https://ca-xxxxx.azurecontainerapps.io`).\n\nThe app defaults to **Git URL mode** — paste any public HTTPS repository URL and click **Verify Codebase**:\n\n```\nhttps://github.com/user/repo\nhttps://github.com/user/repo.git\nhttps://dev.azure.com/org/project/_git/repo\n```\n\nThe server clones the repository into a temporary directory (shallow clone, depth 1, 120 s timeout), runs all verification agents against the Python files, and streams progress back to your browser via Server-Sent Events. The cloned directory is cleaned up automatically after the report is generated.\n\n\u003e **Note:** Only **HTTPS** Git URLs are accepted. SSH URLs (`git@github.com:…`) and `file://` URLs are rejected for security. Private repositories are supported only if the server's environment has Git credentials configured (e.g. via a Git credential helper).\n\nYou can also switch to **Local directory mode** using the toggle in the UI, which is useful when running locally but is not available for cloud-deployed instances (the container does not have access to your local filesystem).\n\n#### Local Docker run (without full Azure deploy)\n\n```bash\ndocker build -t code-verification .\ndocker run -p 8000:8000 --env-file .env code-verification\n```\n\n---\n\n## Scoring Rubrics\n\nEach area is scored from 0 (worst) to 5 (best). The total score is the sum of all four areas (max 20).\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eCode Quality and Coding Standards\u003c/strong\u003e\u003c/summary\u003e\n\n| Score | Meaning |\n|-------|---------|\n| 0 | Code fails linting or contains syntax errors |\n| 1 | Major formatting issues; inconsistent naming; poor structure |\n| 2 | Passes basic linting but contains frequent style violations |\n| 3 | Mostly compliant with coding standards; minor issues |\n| 4 | Fully compliant with coding standards; clean and consistent structure |\n| 5 | Fully compliant, idiomatic, and optimized for readability |\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eFunctional Correctness\u003c/strong\u003e\u003c/summary\u003e\n\n| Score | Meaning |\n|-------|---------|\n| 0 | Core functionality is broken or produces incorrect results |\n| 1 | Major features malfunction; incorrect behavior is common |\n| 2 | Basic functionality works, but edge cases frequently fail |\n| 3 | Core functionality works as intended; minor bugs exist |\n| 4 | Functionality is correct across typical and edge cases |\n| 5 | Functionality is fully correct and robust across all expected scenarios |\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eHandling of Known Errors\u003c/strong\u003e\u003c/summary\u003e\n\n| Score | Meaning |\n|-------|---------|\n| 0 | Known error conditions are not handled and cause crashes or undefined behavior |\n| 1 | Minimal error handling; many known errors propagate unhandled |\n| 2 | Some known errors are handled, but coverage is inconsistent |\n| 3 | Most known error cases are handled with reasonable safeguards |\n| 4 | All known error conditions are explicitly handled with clear recovery or messaging |\n| 5 | Known errors are comprehensively handled with graceful recovery and clear diagnostics |\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cstrong\u003eHandling of Unknown Errors\u003c/strong\u003e\u003c/summary\u003e\n\n| Score | Meaning |\n|-------|---------|\n| 0 | Unexpected errors cause crashes, data corruption, or undefined behavior |\n| 1 | Global error handling exists but provides little protection or visibility |\n| 2 | Some safeguards exist, but unexpected failures are not consistently contained |\n| 3 | Unexpected errors are generally contained and logged without crashing the system |\n| 4 | Robust fallback mechanisms prevent most unknown errors from causing failures |\n| 5 | System is resilient to unknown errors through defensive programming and comprehensive logging |\n\n\u003c/details\u003e\n\n---\n\n## Contributing\n\nContributions are welcome. Please open an issue to discuss proposed changes before submitting a pull request.\n\n## License\n\nThis project is licensed under the [MIT License](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleestott%2Fopustest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fleestott%2Fopustest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleestott%2Fopustest/lists"}