{"id":50332211,"url":"https://github.com/ericchansen/azure-ai-foundry-agent-apis","last_synced_at":"2026-05-29T10:30:50.135Z","repository":{"id":357702977,"uuid":"1238171505","full_name":"ericchansen/azure-ai-foundry-agent-apis","owner":"ericchansen","description":"Tested examples for retrieving agent metadata, operational metrics, and evaluation scores from Azure AI Foundry via Python SDK, REST API, and cURL","archived":false,"fork":false,"pushed_at":"2026-05-13T22:39:04.000Z","size":17,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-05-14T00:56:18.301Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ericchansen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-13T22:05:52.000Z","updated_at":"2026-05-13T22:39:20.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ericchansen/azure-ai-foundry-agent-apis","commit_stats":null,"previous_names":["ericchansen/azure-ai-foundry-agent-apis"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/ericchansen/azure-ai-foundry-agent-apis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ericchansen%2Fazure-ai-foundry-agent-apis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ericchansen%2Fazure-ai-foundry-agent-apis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ericchansen%2Fazure-ai-foundry-agent-apis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ericchansen%2Fazure-ai-foundry-agent-apis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ericchansen","download_url":"https://codeload.github.com/ericchansen/azure-ai-foundry-agent-apis/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ericchansen%2Fazure-ai-foundry-agent-apis/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33648529,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-29T02:00:06.066Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-29T10:30:49.601Z","updated_at":"2026-05-29T10:30:50.130Z","avatar_url":"https://github.com/ericchansen.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Azure AI Foundry — Agent APIs \u0026 Operational Metrics Guide\n\nPractical, tested examples for retrieving **agent metadata**, **per-run token usage**, **aggregate operational metrics**, and **evaluation scores** from [Azure AI Foundry](https://learn.microsoft.com/en-us/azure/ai-foundry/).\n\n## The Problem\n\nAzure AI Foundry's UI shows rich agent details — name, model, tools, versioning, token usage, error rates, latency — but the documentation doesn't clearly explain **how to access all of this programmatically**. This repo fills that gap with tested code samples covering every API surface.\n\n## Quick-Reference: Where to Get What\n\n| Data You Need | API Surface | Example |\n|---|---|---|\n| Agent name, model, tools, instructions, metadata | OpenAI Assistants API | [`01_agent_metadata.py`](examples/01_agent_metadata.py) |\n| Agent versioning \u0026 lifecycle | Foundry Project v1 API | [`01_agent_metadata.py`](examples/01_agent_metadata.py) |\n| Per-run token usage \u0026 latency | Runs API (per-thread) | [`02_runs_and_usage.py`](examples/02_runs_and_usage.py) |\n| Run steps (tool calls, sub-steps) | Run Steps API | [`02_runs_and_usage.py`](examples/02_runs_and_usage.py) |\n| **Aggregate metrics** (error rates, total tokens, avg latency) | **Application Insights** via KQL | [`03_tracing_and_metrics.py`](examples/03_tracing_and_metrics.py) |\n| Evaluation scores (coherence, safety, etc.) | OpenAI Evals API | [`04_evaluations.py`](examples/04_evaluations.py) |\n| All of the above via cURL | REST | [`curl_reference.sh`](examples/curl_reference.sh) |\n\n\u003e **Key finding**: There is **no direct \"metrics endpoint\"** for aggregate operational data. The AI Foundry monitoring dashboard reads exclusively from **Application Insights** via OpenTelemetry traces.\n\n## Architecture\n\n```\n┌──────────────────────┐          ┌─────────────────────────────┐\n│  Your Application     │          │  Azure AI Foundry            │\n│                       │          │                              │\n│  1. List agents       │── REST ─▶│  /openai/assistants          │\n│  2. Get agent detail  │          │  /openai/threads/*/runs      │\n│  3. Get run usage     │          │  /openai/threads/*/runs/steps│\n│                       │          │                              │\n│  4. Aggregate metrics │── KQL ──▶│  Application Insights        │\n│     (token, latency,  │          │  (via Azure Monitor Logs)    │\n│      error rates)     │          │                              │\n│  5. Eval scores       │── REST ─▶│  OpenAI Evals API            │\n└──────────────────────┘          └─────────────────────────────┘\n```\n\n## Prerequisites\n\n1. **Azure AI Foundry project** with an AI Services resource and at least one model deployment\n2. **Python 3.9+**\n3. **Azure CLI** (`az login`) or a service principal for authentication\n4. (For metrics) **Application Insights** connected to your Foundry project\n\n## Setup\n\n```bash\n# Clone the repo\ngit clone https://github.com/ericchansen/azure-ai-foundry-agent-apis.git\ncd azure-ai-foundry-agent-apis\n\n# Install dependencies\npip install -r requirements.txt\n\n# Configure environment\ncp .env.example .env\n# Edit .env with your values — see comments in the file\n```\n\n## Examples\n\n### 1. Agent Metadata ([`examples/01_agent_metadata.py`](examples/01_agent_metadata.py))\n\nList agents, retrieve details (model, tools, instructions, temperature), and query agent versions.\n\n```bash\npython examples/01_agent_metadata.py\n```\n\n### 2. Runs \u0026 Token Usage ([`examples/02_runs_and_usage.py`](examples/02_runs_and_usage.py))\n\nCreate a thread, run an agent, poll for completion, then extract per-run token usage, latency, and run steps.\n\n```bash\npython examples/02_runs_and_usage.py\n```\n\n**Sample run object** (what you get back):\n```json\n{\n  \"status\": \"completed\",\n  \"model\": \"gpt-4o-mini\",\n  \"usage\": {\n    \"prompt_tokens\": 455,\n    \"completion_tokens\": 103,\n    \"total_tokens\": 558\n  },\n  \"started_at\": 1778707463,\n  \"completed_at\": 1778707465\n}\n```\n\n### 3. Tracing \u0026 Aggregate Metrics ([`examples/03_tracing_and_metrics.py`](examples/03_tracing_and_metrics.py))\n\nEnable OpenTelemetry tracing, send agent call telemetry to Application Insights, then query aggregate metrics with KQL.\n\n```bash\npython examples/03_tracing_and_metrics.py\n```\n\n### 4. Evaluations ([`examples/04_evaluations.py`](examples/04_evaluations.py))\n\nCreate and run evaluations using the OpenAI Evals API (`score_model` evaluator), upload test data, poll for completion, retrieve per-item scores.\n\n```bash\npython examples/04_evaluations.py\n```\n\n### cURL Reference ([`examples/curl_reference.sh`](examples/curl_reference.sh))\n\nComplete REST API reference using cURL — useful for non-Python integrations or quick testing.\n\n## API Authentication\n\nAll examples use [DefaultAzureCredential](https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential), which supports:\n\n- **Azure CLI** (`az login`) — best for development\n- **Managed Identity** — best for production\n- **Service Principal** — best for CI/CD\n\nThe REST APIs accept either:\n- **Bearer token** (scope: `https://cognitiveservices.azure.com/.default`)\n- **API key** via the `api-key` header\n\n## Key OpenTelemetry Attributes\n\nWhen tracing is enabled, these attributes appear in Application Insights and can be queried with KQL:\n\n| Attribute | Purpose |\n|---|---|\n| `gen_ai.agent.id` | Agent identifier |\n| `gen_ai.agent.name` | Human-readable name |\n| `gen_ai.usage.input_tokens` | Prompt token count |\n| `gen_ai.usage.output_tokens` | Completion token count |\n| `gen_ai.request.model` | Model used (e.g., `gpt-4o-mini`) |\n| `gen_ai.operation.name` | `invoke_agent`, `chat`, `execute_tool` |\n| `gen_ai.response.finish_reasons` | `[\"stop\"]`, `[\"error\"]`, etc. |\n\n## Evaluators\n\nThe OpenAI Evals API surface supports these evaluator types:\n\n| Type | Purpose |\n|---|---|\n| `score_model` | LLM-as-judge scoring (1–5 scale, custom prompt) — **tested in this repo** |\n| `label_model` | LLM-as-judge classification (pass/fail, custom labels) |\n| `text_similarity` | Fuzzy text matching |\n| `string_check` | Exact/substring match |\n| `python` | Custom Python evaluation function |\n| `endpoint` | Call an external evaluation endpoint |\n\n\u003e **Note:** The Azure-specific `azure_ai_evaluator` type (`builtin.coherence`, `builtin.task_adherence`, etc.) is available via the Foundry portal and [Azure AI Evaluation SDK](https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/develop/cloud-evaluation), but is not supported on the OpenAI Evals REST surface as of `2025-04-01-preview`.\n\n## RBAC Requirements\n\n| Scenario | Role Needed | Scope |\n|---|---|---|\n| Query App Insights metrics | **Log Analytics Reader** | App Insights resource + linked workspace |\n| Continuous evaluation | **Azure AI User** | Foundry project (assigned to project managed identity) |\n| Agent CRUD / Runs | **Cognitive Services User** | AI Services resource |\n\n## Documentation References\n\n| Topic | URL |\n|---|---|\n| AI Projects Python SDK | [learn.microsoft.com](https://learn.microsoft.com/en-us/python/api/overview/azure/ai-projects-readme) |\n| AI Agents Python SDK | [learn.microsoft.com](https://learn.microsoft.com/en-us/python/api/azure-ai-agents/azure.ai.agents.agentsclient) |\n| Foundry Project REST API | [learn.microsoft.com](https://learn.microsoft.com/en-us/azure/ai-foundry/reference/foundry-project-rest-preview) |\n| Agent Tracing (OpenTelemetry) | [learn.microsoft.com](https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/develop/trace-agents-sdk) |\n| Agent Monitoring Dashboard | [learn.microsoft.com](https://learn.microsoft.com/en-us/azure/ai-foundry/observability/how-to/how-to-monitor-agents-dashboard) |\n| Cloud Evaluation SDK | [learn.microsoft.com](https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/develop/cloud-evaluation) |\n\n## Known Limitations\n\n- **No cross-agent \"all runs\" endpoint** — the Runs API is per-thread only. Use Application Insights KQL for aggregate views.\n- **Tracing is experimental** — the `AZURE_EXPERIMENTAL_ENABLE_GENAI_TRACING=true` feature gate means span attributes may change in future releases.\n- **Trace ingestion delay** — newly sent traces take 2–5 minutes to appear in Application Insights queries.\n- **Foundry v1 API** requires agents created through the Foundry portal or v1 API; agents created via the OpenAI Assistants API appear only on the OpenAI surface.\n\n## License\n\n[MIT](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fericchansen%2Fazure-ai-foundry-agent-apis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fericchansen%2Fazure-ai-foundry-agent-apis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fericchansen%2Fazure-ai-foundry-agent-apis/lists"}