{"id":22138159,"url":"https://github.com/microsoft/azureopenai-with-apim","last_synced_at":"2025-04-05T13:06:56.116Z","repository":{"id":194375105,"uuid":"690700274","full_name":"microsoft/AzureOpenAI-with-APIM","owner":"microsoft","description":"Deploy APIM. Auto-configure it to work with your Azure Open AI.","archived":false,"fork":false,"pushed_at":"2024-12-03T15:16:20.000Z","size":12333,"stargazers_count":96,"open_issues_count":3,"forks_count":41,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-04-05T13:06:29.643Z","etag":null,"topics":["api","api-management","apim","azure","azure-government","azure-openai","federal","government","open-ai","zero-trust"],"latest_commit_sha":null,"homepage":"","language":"Bicep","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/microsoft.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":"SUPPORT.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-12T17:39:40.000Z","updated_at":"2025-04-04T01:57:01.000Z","dependencies_parsed_at":"2024-03-14T16:27:13.725Z","dependency_job_id":"168ddc64-bc28-4158-9291-64db70145b87","html_url":"https://github.com/microsoft/AzureOpenAI-with-APIM","commit_stats":null,"previous_names":["microsoft/azureopenai-with-apim"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FAzureOpenAI-with-APIM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FAzureOpenAI-with-APIM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FAzureOpenAI-with-APIM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FAzureOpenAI-with-APIM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/microsoft","download_url":"https://codeload.github.com/microsoft/AzureOpenAI-with-APIM/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247339156,"owners_count":20923014,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","api-management","apim","azure","azure-government","azure-openai","federal","government","open-ai","zero-trust"],"created_at":"2024-12-01T20:08:45.384Z","updated_at":"2025-04-05T13:06:56.089Z","avatar_url":"https://github.com/microsoft.png","language":"Bicep","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003e [!NOTE]\n\u003e\n\u003e Updated on 2024-12-03, \n\u003e\n\u003e - Added Priority routing policy\n\u003e   - This provides ability to include a Priority value in the HTTP header, which will route the request to your predefined backend. You can combine this with other policies like rate throttling to control throughput while ensuring defined services meet organizational SLAs. Combine with the circuit breaker pool for requests that do not include priority header info. \n\u003e - Added more details and reporting on Cost Management\n\u003e   - Details on implementing the policies and how to generate log analytics reports via App insights\n\u003e   - Details on creating Power BI reports using log analytics as a data source, this can provide leadership and management access to reporting without needing access to Azure Portal.\n\n\n\n# Manage Azure OpenAI using APIM\n\n## Overview\n\nOne-button deploy APIM, Key vault, and Log Analytics. Auto-configure APIM to work with your Azure OpenAI endpoint. \n\nUsing Azure's APIM orchestration provides a organizations with a powerful way to scale and manage their Azure OpenAI service without deploying Azure OpenAI endpoints everywhere. Administrators can issue subscription keys via APIM for accessing a single Azure OpenAI service instead of having teams share Azure OpenAI keys. APIM delivers usage metrics along with API monitoring to improve business intelligence. APIM policies control access, throttling, and a mechanism for chargeback cost models.\n\nGain conceptual details and technical step-by-step knowledge on implementing APIM to support your Azure OpenAI resiliency, scalability, performance, monitoring, and charge back capabilities. \n\n\u003e [!IMPORTANT]\n\u003e\n\u003e The goal with this repo is to provide more than conceptual knowledge on the services and technology. We want you to walk away from this repo knowing EXACTLY what to do to add these capabilities to YOUR enterprise. If we are missing information  or steps or details or visuals or videos or anything that adds friction to you gaining this knowledge effectively and efficiently, please say something - [Add your Ideas · Discussions · GitHub](https://github.com/microsoft/AzureOpenAI-with-APIM/discussions/categories/ideas).\n\n## Solutions\n\nThere are two solutions developed to meet the needs of the organization from a sandbox to model a production environment.\n\n\u003e [!NOTE]\n\u003e\n\u003e With Azure OpenAI's availability in Azure Government, the solutions are consolidated down because they are identical whether deployed into Azure Commercial or Azure Government.\n\n- [API Management to Azure OpenAI](#api-management-to-azure-openai)\n- [API Management to Azure OpenAI with private endpoints](#api-management-to-azure-openai-with-private-endpoints)\n\n## Using it\n\nOnce the service is deployed, use the following section to understand how to access your Azure OpenAI service via APIM.\n\n- [Steps and examples on how to use your APIM-based API to query your Azure OpenAI API](#tying-it-all-together)\n\n### Utilization per subscription (support Charge-back)\n\n\u003e [!TIP]\n\u003e\n\u003e A subscription, in the context of APIM, is the authorization of a user or group to access an API.\n\nCalculate token counts per subscription. This can be helpful to monitor utilization of Azure OpenAI but also provides the foundation for a charge-back model if, internally, your organization provides a shared services model where Azure OpenAI is deployed and provided back to the business with one of the following models:\n\n- For free and the team managing the service pays for all costs\n- Cost basis so the team managing the service provides access to said service but requires the internal business consumer to pay for its use. \n\nUse the following link to deploy an APIM policy and supporting services to capture token counts by subscription using an Event hub, process them using an Azure Function, and log them to Log Analytics workspace for reporting.\n\n- [Deploy service to monitor Azure OpenAI token count per subscription](#azure-openai-token-count-per-subscription)\n\n### Resiliency (Retry Policy)\n\nResiliency is the ability of the service to recover from an issue. With regard to Azure OpenAI and APIM, this means using a Retry Policy to attempt the prompt again. Typically the prompt fails because at the time the user submitted their prompt, the Azure OpenAI endpoint was maxed out of either Tokens Per Minute or Requests Per Minute based on the endpoints quota, which triggers a HTTP 429 error.\n\n- Retry policy to leverage two or more Azure OpenAI endpoints\n- Expands capacity without impact to user experience or requesting increase to existing Azure OpenAI endpoints\n\nSingle region Retry policy means APIM will wait for a specific period of time and attempt to submit the prompt to the same Azure OpenAI endpoint. This is ok for development phases of solutions but not ideal for production.\n\nMulti region Retry policy means APIM will immediately submit the prompt to a completely separate Azure OpenAI endpoint that is deployed to a separate Azure region. This effectively doubles your Azure OpenAI quota and provides resiliency. You can tie this together with the APIM Load Balancer capability (as of April 4, 2024, this is Azure Commercial only and in Preview) you can have scale and resiliency. This is discussed in more detail [LINK HERE]\n\n- [Implement Resiliency](#resiliency)\n\n### Scalability (Load Balancing)\n\nScalability provides the ability of your Azure OpenAI service to support higher loads without necessarily increasing regional quotas. This feature  (as of April 4, 2024, this is Azure Commercial only and in Preview) uses two or more Azure OpenAI endpoints in a round-robin load balancing configuration. This feature is not built into the One-button deploy but perspective implementation is provided so that organizations can implement.\n\nYou can tie this together with the Retry policy (as of April 4, 2024, this is Azure Commercial only and in Preview) you can have scale and resiliency. This is discussed in more detail [LINK HERE]\n\n- [Implement Scalability](#scalability)\n\n### Performance (Provisioned Throughput Units)\n\nScale provides ability of an organization to support higher loads by leveraging multiple regions but the TPM cost model is a best effort compute with no SLAs. When using TPM pay as you go model, as long as your Azure OpenAI endpoint has quota - your prompts will be processed but their latency may be higher than anticipated and variability may be more inconsistent than anticipated.\n\nTo improve performance, Azure OpenAI has a cost model called Provisioned Throughput Units (PTU). When using PTUs, the organization is procuring an allotted amount of GPU to process their models. No other organization or individual can use that GPU. This has a positive effect of reducing latency and tightening up the variability in latency. It has a secondary effect of improving [cost forecasting](#cost-forecasting-provisioned-throughput-units-+-rate-limiting) (discussed in more detail further in this article). \n\n- [Implement Performance](#performance)\n\n#### Implementing Priority-based Routing\n\nTo effectively manage performance and resource utilization, you can implement priority-based routing in Azure API Management (APIM). This allows you to direct high-priority requests to more performant (and potentially more costly) PTU endpoints, while routing lower-priority requests to standard pay-as-you-go endpoints.\n\n#### APIM Policy Configuration\n\nBelow is an example of an APIM policy that routes requests based on the `Priority` header value:\n\n```xml\n\u003cpolicies\u003e\n    \u003cinbound\u003e\n        \u003cbase /\u003e\n        \u003c!-- Check if the Priority header exists --\u003e\n        \u003cset-variable name=\"priorityValue\" value=\"@(context.Request.Headers.GetValueOrDefault(\"Priority\", \"0\"))\" /\u003e\n\n        \u003c!-- Convert the priorityValue to an integer --\u003e\n        \u003cset-variable name=\"priorityInt\" value=\"@{\n            int priority;\n            return int.TryParse((string)context.Variables[\"priorityValue\"], out priority) ? priority : 0;\n        }\" /\u003e\n\n        \u003c!-- Conditional routing based on priorityInt --\u003e\n        \u003cchoose\u003e\n            \u003c!-- Route to PTU Endpoint if Priority is between 0 and 100 --\u003e\n            \u003cwhen condition=\"@(context.Variables.GetValueOrDefault\u003cint\u003e(\"priorityInt\") \u003e= 0 \u0026\u0026 context.Variables.GetValueOrDefault\u003cint\u003e(\"priorityInt\") \u003c= 100)\"\u003e\n                \u003cset-backend-service backend-id=\"aoai-ptu-backend\" /\u003e\n            \u003c/when\u003e\n            \u003c!-- Route to Regional Endpoint if Priority is between 101 and 500 --\u003e\n            \u003cwhen condition=\"@(context.Variables.GetValueOrDefault\u003cint\u003e(\"priorityInt\") \u003e= 101 \u0026\u0026 context.Variables.GetValueOrDefault\u003cint\u003e(\"priorityInt\") \u003c= 500)\"\u003e\n                \u003cset-backend-service backend-id=\"aoai-regional-backend\" /\u003e\n            \u003c/when\u003e\n            \u003c!-- Route to Data Zone Deployment if Priority is between 501 and 1000 --\u003e\n            \u003cwhen condition=\"@(context.Variables.GetValueOrDefault\u003cint\u003e(\"priorityInt\") \u003e= 501 \u0026\u0026 context.Variables.GetValueOrDefault\u003cint\u003e(\"priorityInt\") \u003c= 1000)\"\u003e\n                \u003cset-backend-service backend-id=\"aoai-data_zone-backend\" /\u003e\n            \u003c/when\u003e\n            \u003c!-- Default routing using Circuit Breaker pool if Priority header is missing or out of range --\u003e\n            \u003cotherwise\u003e\n                \u003cset-backend-service backend-id=\"aoai-circuit_breaker-backend\" /\u003e\n            \u003c/otherwise\u003e\n        \u003c/choose\u003e\n    \u003c/inbound\u003e\n    \u003cbackend\u003e\n        \u003cbase /\u003e\n    \u003c/backend\u003e\n    \u003coutbound\u003e\n        \u003cbase /\u003e\n    \u003c/outbound\u003e\n    \u003con-error\u003e\n        \u003cbase /\u003e\n    \u003c/on-error\u003e\n\u003c/policies\u003e\n```\n\n**Explanation:**\n\n- **Set Variable `priorityValue`:** Retrieves the `Priority` header value from the incoming request. If it's missing, it defaults to `\"0\"`.\n- **Set Variable `priorityInt`:** Converts the `priorityValue` to an integer for comparison purposes.\n- **Conditional Routing with `\u003cchoose\u003e`:**\n  - **First `\u003cwhen\u003e` Condition:** Routes to the PTU Endpoint if `Priority` is between **0 and 100**.\n  - **Second `\u003cwhen\u003e` Condition:** Routes to the Regional Endpoint if `Priority` is between **101 and 500**.\n  - **Third `\u003cwhen\u003e` Condition:** Routes to the Data zone Deployment if `Priority` is between **501 and 1000**.\n  - **`\u003cotherwise\u003e` Condition:** Routes to a default endpoint using a Circuit Breaker pool if the `Priority` header is missing or doesn't match any of the defined conditions.\n\n**Notes:**\n\n- Replace the `base-url` in `\u003cset-backend-service\u003e` with your actual endpoint URLs or backend name.\n- Ensure that the priority ranges are correctly defined and non-overlapping.\n- This policy should be added to the inbound section of your APIM policy configuration.\n\n#### HTTP Header Configuration\n\nTo make this routing work, clients need to include the `Priority` header in their HTTP requests.\n\n- **HTTP Request Header:**\n  - **Header Name:** `Priority`\n  - **Header Value:** An integer between **0 and 1000** depending on the desired priority level.\n\n**Example HTTP Request:**\n\n```http\nPOST /apim/url HTTP/1.1\nHost: your-apim-instance.azure-api.net\nPriority: 50\nContent-Type: application/json\nAuthorization: Bearer your-access-token\n\n{\n    \"your\": \"request-body\"\n}\n```\n\n### Cost Management (Rate Limiting)\n\n- Provide cost management per subscription with Rate Throttling\n- [Implement Cost Management](#cost-management)\n\n### Cost Forecasting (Provisioned Throughput Units + Rate Limiting)\n\n- [Implement Cost Forecasting](#cost-forecasting)\n\n### Access Control\n\n#### Managed Identity\n\n**APIM to Azure OpenAI**\n\n\u003e [!IMPORTANT]\n\u003e\n\u003e The latest update to this repo moves to Managed Identity for the one-button deployments and guides for modifying existing APIM services.\n\nManaged identities is the ideal method for authenticating APIM to Azure OpenAI. This eliminates the need to manage the rotation of keys and for keys to be stored anywhere beyond the Azure OpenAI endpoint. \n\n- [Implement authenticating APIM to Azure OpenAI using Managed Identity](#apim-to-azure-openai)\n\n**Client or Application to APIM**\n\nManaged identities of the Azure App Service, Virtual Machine, Azure Kubernetes Service, or any other compute service in Azure is the preferred method of authentication. It improves security and eliminates the issuance of keys and tokens. Clients can use an OAuth token if required to validate access when using Azure OpenAI via APIM.\n\nThese methods shifts the burden of authentication from the application and onto APIM, which improves performance, scalability, operational management, and identity service selection. Update APIM's identity provider for OAuth and that update flows down to the application without any modification to the application.  \n\nThe examples provide both methods along with guidance on how to setup the client or application to run the code so that managed identity technique can be used.\n\n- [Implement authenticating application to APIM using Managed Identity](#application-to-apim)\n- [Implement authenticating client to APIM using Entra](#client-to-apim)\n\n#### Tokens and Keys\n\n**APIM to Azure OpenAI SAS Token**\n\nSubscription key is like a SAS token provided by the backend service but it is issued by APIM for use by the user or group. These are fine during the development phase. We've retained the policy xml file in the repo as an example but this technique is not used.\n\n- [Implement authenticating APIM to Azure OpenAI using SAS](#apim-to-azure-openai-sas-token)\n\n**Client or Application to APIM**\n\nFor development and when running test commands from your workstation, using the Subscription key is straightforward but we recommend moving from the Subscription key to Managed Identities beyond development. The examples provide both methods along with guidance on how to setup the client or application to run the code so that managed identity technique can be used.\n\n- [Implement authenticating application to APIM using Subscription Key](#application-to-apim-subscription-key)\n- [Implement authenticating client to APIM using Subscription Key](#client-to-apim-subscription-key)\n\n## Prerequisites\n\n### Azure \n\n- Contributor permissions to subscription or resource group\n- Resource Group (or ability to create)\n- Azure OpenAI service deployed\n  - [How-to: Create and deploy an Azure OpenAI Service resource - Azure OpenAI | Microsoft Learn](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal)\n  - If using Multi-region, then deploy an additional Azure OpenAI service in a different region.\n- Azure OpenAI model deployed\n  - [How-to: Create and deploy an Azure OpenAI Service resource - Azure OpenAI | Microsoft Learn](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal#deploy-a-model)\n  - If using Multi-region, make sure Deployment names are identical.\n- Azure OpenAI service URL\n  - [Quickstart - Deploy a model and generate text using Azure OpenAI Service - Azure OpenAI | Microsoft Learn](https://learn.microsoft.com/en-us/azure/ai-services/openai/quickstart?tabs=command-line\u0026pivots=rest-api#retrieve-key-and-endpoint)\n  - If using Multi-region, collect the additional Azure OpenAI service URL.\n\n\n### One-Button\n\nEach solution provides a simple one-button deployment. Select the \"Deploy to Azure\" button which will open the Azure portal and provide a form for details.\n\n### Command Line\n\nTo use the command line deployment method, fork the library and use Codespaces or clone the forked library to your local computer.\n\n- [GitHub Codespaces](https://github.com/features/codespaces)\n\n#### Azure CLI\n\n- [How to install the Azure CLI | Microsoft Learn](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli)\n- [Connect to Azure Government with Azure CLI - Azure Government | Microsoft Learn](https://learn.microsoft.com/en-us/azure/azure-government/documentation-government-get-started-connect-with-cli)\n\n#### Azure PowerShell\n\n- [How to install Azure PowerShell | Microsoft Learn](https://learn.microsoft.com/en-us/powershell/azure/install-azure-powershell?view=azps-10.2.0)\n- [Connect to Azure Government with PowerShell - Azure Government | Microsoft Learn](https://learn.microsoft.com/en-us/azure/azure-government/documentation-government-get-started-connect-with-ps)\n\n## Architectures\n\nThe following architectural solutions support two use-cases in the Azure Commercial and Azure Government environments. Determining which solution to implement requires understanding of your current utilization of Azure. \n\n- [API Management to Azure OpenAI](#api-management-to-azure-open-ai)\n  - Supports Azure Commercial and Azure Government\n  - Developing proof of concept or minimum viable production solution.\n  - *Isolated from enterprise networking* using internal networks, Express Routes, and site-2-site VPN connections from the cloud to on-premises networks.\n  - Assigns APIM the \n- [API Management to Azure OpenAI with private endpoints](#api-management-to-azure-open-ai-with-private-endpoints)\n  - Supports Azure Commercial and Azure Government\n  - Pilot or production solution.\n  - *Connected to the enterprise networking* using internal networks, Express Routes, and site-2-site VPN connections from the cloud to on-premises networks.\n\n\n### API Management to Azure OpenAI\n\nUse API management deployed to your Azure environment using public IP addresses for accessing APIM and for APIM to access the Azure OpenAI API. Access to the services is secured using keys and Defender for Cloud. \n\n\u003e [!NOTE]\n\u003e\n\u003e Only API Management Service is deployed, this solution requires the Azure OpenAI service to already exist.\n\n![API Management to Azure OpenAI](./images/apim-to-aoai.png)\n\n1. Users and Groups are used to assign access to an API using subscriptions.\n   1. Each User or Group can be assigned their own policies like rate-limiting to control use\n2. The Azure OpenAI API uses policies to assign backends, retry, rate-throttling, and token counts.\n3. The backends are assigned using a policy and can include load balance groups. Retry policies reference additional backends for resiliency.\n4. One or more Azure OpenAI service (endpoint) can be used to manage scale and resiliency.\n   1. The endpoints will reside in different regions so that they can utilize the maximum quota available to them.\n5. Policies are used to collect information, perform actions, and manipulate user connections.\n6. App Insights are used to create dashboards to monitor performance and use.\n\n#### Deploy\n\n**! NOTE !** - It can take up to 45 minutes for all services to deploy as API Management has many underlying Azure resources deployed running the service.\n\n##### One Button\n\nSimple one-button deployment, opens in Azure Portal\n\n[![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fmicrosoft%2FAzureOpenAI-with-APIM%2Fmain%2Fpublic-apim.json)\n\n[![Deploy to Azure Government](https://aka.ms/deploytoazuregovbutton)](https://portal.azure.us/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fmicrosoft%2FAzureOpenAI-with-APIM%2Fmain%2Fpublic-apim.json)\n\n##### Azure CLI\n\n```powershell\n# Update the following variables to use the appropriate resource group and subscription.\n$resourceGroupName = \"RG-APIM-OpenAI\"\n$location = \"East US\" # Use MAG region when deploying to MAG\n$subscriptionName = \"MySubscription\"\n\n# az cloud set --name AzureUSGovernment # Uncomment when deploying to MAG\naz login\naz account set --subscription $subscriptionName\naz group create --name $resourceGroupName --location $location\naz deployment group create --resource-group $resourceGroupName  --template-file .\\public-apim.bicep --mode Incremental\n```\n\n##### Azure PowerShell\n\n```powershell\n# Update the following variables to use the appropriate resource group and subscription.\n$resourceGroupName = \"RG-APIM-OpenAI\"\n$location = \"East US\" # Use MAG region when deploying to MAG\n$subscriptionName = \"MySubscription\"\n\nConnect-AzAccount #-Environment AzureUSGovernment # Uncomment when deploying to MAG\nSet-AzContext -Subscription $subscriptionName\nNew-AzResourceGroup -Name $resourceGroupName -Location $location\nNew-AzResourceGroupDeployment -ResourceGroupName $resourceGroupName -TemplateFile .\\public-apim.bicep -Verbose -mode Incremental\n```\n\n#### Next steps\n\n- Now that APIM is deployed and automatically configured to work with your Azure OpenAI service\n  - [Click here to learn how do you use the APIM endpoint to interact with Azure OpenAI?](#tying-it-all-together)\n\n### API Management to Azure OpenAI with private endpoints\n\nUse API management deployed to your Azure environment using private IP addresses for accessing APIM and for APIM to access the Azure OpenAI API. Access to the services is secured using private network connectivity, keys, and Defender for Cloud. Access to the private network is controlled by customer infrastructure and supports internal routing via Express Route or site-2-site VPN for broader enterprise network access like on-premises data centers or site-based users.\n\n![API Management to Azure OpenAI with private endpoints](./images/apim-to-aoai-with-private-endpoints.png)\n\n1. Users and Groups are used to assign access to an API using subscriptions.\n   1. Each User or Group can be assigned their own policies like rate-limiting to control use\n2. The Azure OpenAI API uses policies to assign backends, retry, rate-throttling, and token counts.\n3. The backends are assigned using a policy and can include load balance groups. Retry policies reference additional backends for resiliency.\n4. One or more Azure OpenAI service (endpoint) can be used to manage scale and resiliency. \n   1. The endpoints will reside in different regions so that they can utilize the maximum quota available to them.\n5. Policies are used to collect information, perform actions, and manipulate user connections.\n6. App Insights are used to create dashboards to monitor performance and use.\n\n#### Deploy\n\n**! NOTE !** - It can take up to 45 minutes for all services to deploy as API Management has many underlying Azure resources deployed running the service.\n\n##### One Button\n\nSimple one-button deployment, opens in Azure Portal\n\n[![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fmicrosoft%2FAzureOpenAI-with-APIM%2Fmain%2Fprivate-apim.json)\n\n[![Deploy to Azure Government](https://aka.ms/deploytoazuregovbutton)](https://portal.azure.us/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fmicrosoft%2FAzureOpenAI-with-APIM%2Fmain%2Fprivate-apim.json)\n\n##### Azure CLI\n\n```powershell\n# Update the following variables to use the appropriate resource group and subscription.\n$resourceGroupName = \"RG-APIM-OpenAI\"\n$location = \"East US\" # Use MAG region when deploying to MAG\n$subscriptionName = \"MySubscription\"\n\n# az cloud set --name AzureUSGovernment # Uncomment when deploying to MAG\naz login\naz account set --subscription $subscriptionName\naz group create --name $resourceGroupName --location $location\naz deployment group create --resource-group $resourceGroupName  --template-file .\\private-apim.bicep --mode Incremental\n```\n\n##### Azure PowerShell\n\n```powershell\n# Update the following variables to use the appropriate resource group and subscription.\n$resourceGroupName = \"RG-APIM-OpenAI\"\n$location = \"East US\" # Use MAG region when deploying to MAG\n$subscriptionName = \"MySubscription\"\n\nConnect-AzAccount #-Environment AzureUSGovernment # Uncomment when deploying to MAG\nSet-AzContext -Subscription $subscriptionName\nNew-AzResourceGroup -Name $resourceGroupName -Location $location\nNew-AzResourceGroupDeployment -ResourceGroupName $resourceGroupName -TemplateFile .\\private-apim.bicep -Verbose -mode Incremental\n```\n\n#### Next steps\n\n- Now that APIM is deployed and automatically configured to work with your Azure OpenAI service\n  - [Click here to learn how do you use the APIM endpoint to interact with Azure OpenAI?](#tying-it-all-together)\n\n## Azure OpenAI token count per subscription\n\nTBD\n\nPolicy for collecting tokens and user id\n\n### Log Analytics workspace\n\nTBD\n\n### Event Hub\n\nTBD\n\n### PowerApps Dashboard\n\nTBD\n\n#### Demo\n\nTBD\n\n## Resiliency\n\n#### Retry Policy\n\nTBD\n\n[Azure API Management policy reference - retry | Microsoft Learn](https://learn.microsoft.com/en-us/azure/api-management/retry-policy)\n\n#### Multi-region\n\nTBD\n\n[Ensure reliability of your Azure API Management instance - Azure API Management | Microsoft Learn](https://learn.microsoft.com/en-us/azure/api-management/high-availability)Throttling\n\nTBD\n\n[Advanced request throttling with Azure API Management | Microsoft Learn](https://learn.microsoft.com/en-us/azure/api-management/api-management-sample-flexible-throttling)\n\n## Scalability\n\n- Preview feature for two or more Azure OpenAI endpoints using round-robin load balancing\n- Pair with Resiliency for highly scalable solution\n\n## Performance\n\n- Preview feature for two or more Azure OpenAI endpoints using round-robin load balancing\n- Pair with Resiliency for highly scalable solution\n\n## Cost Management\n\n- Provide cost management per subscription\n\nTo effectively manage the costs associated with Azure OpenAI Service usage, you can implement policies in Azure API Management (APIM) that control and monitor the number of tokens consumed by each request. By limiting tokens per subscription and emitting detailed metrics, you can enforce usage quotas, prevent overuse, and enable charge-back models for cost recovery.\n\n### Implementing Token Limits and Metrics Emission\n\nThe following APIM policy helps you manage costs by:\n\n- Limiting the number of tokens a subscription can consume per minute.\n- Estimating prompt tokens to include both prompt and completion tokens in the limit.\n- Emitting token usage metrics with dimensions that help you analyze and report on token consumption per deployment and subscription.\n\n#### APIM Policy Configuration\n\n```\nxmlCopy code\u003cpolicies\u003e\n    \u003cinbound\u003e\n        \u003c!-- Set the backend service to your Azure OpenAI endpoint --\u003e\n        \u003cset-backend-service id=\"apim-generated-policy\" backend-id=\"azure-openai-openai-endpoint\" /\u003e\n\n        \u003c!-- Extract the deployment ID from the URL path after '/deployments/' --\u003e\n        \u003cset-variable name=\"deploymentId\" value=\"@(context.Request.Url.Path.Split('/').ElementAtOrDefault(3))\" /\u003e\n\n        \u003c!-- Limit tokens per minute per subscription --\u003e\n        \u003cazure-openai-token-limit\n            tokens-per-minute=\"10000000\"\n            counter-key=\"@(context.Subscription.Id)\"\n            estimate-prompt-tokens=\"true\"\n            tokens-consumed-header-name=\"consumed-tokens\"\n            remaining-tokens-header-name=\"remaining-tokens\" /\u003e\n\n        \u003c!-- Emit token metrics with custom dimensions --\u003e\n        \u003cazure-openai-emit-token-metric\u003e\n            \u003cdimension name=\"API ID\" /\u003e\n            \u003cdimension name=\"Subscription ID\" /\u003e\n            \u003cdimension name=\"User ID\" /\u003e\n            \u003cdimension name=\"Product ID\" /\u003e\n            \u003c!-- Include the deployment ID as a custom dimension --\u003e\n            \u003cdimension name=\"Deployment ID\" value=\"@(context.Variables.GetValueOrDefault\u003cstring\u003e(\"deploymentId\", \"unknown\"))\" /\u003e\n        \u003c/azure-openai-emit-token-metric\u003e\n\n        \u003c!-- Authenticate using Managed Identity --\u003e\n        \u003cauthentication-managed-identity resource=\"https://cognitiveservices.azure.com/\" /\u003e\n        \u003cbase /\u003e\n    \u003c/inbound\u003e\n    \u003cbackend\u003e\n        \u003cbase /\u003e\n    \u003c/backend\u003e\n    \u003coutbound\u003e\n        \u003cbase /\u003e\n    \u003c/outbound\u003e\n    \u003con-error\u003e\n        \u003cbase /\u003e\n    \u003c/on-error\u003e\n\u003c/policies\u003e\n```\n\n#### Explanation\n\n- **Set Backend Service**: The `\u003cset-backend-service\u003e` element directs the request to your Azure OpenAI endpoint.\n\n  ```\n  xml\n  \n  \n  Copy code\n  \u003cset-backend-service id=\"apim-generated-policy\" backend-id=\"azure-openai-openai-endpoint\" /\u003e\n  ```\n\n- **Extract Deployment ID**: The `\u003cset-variable\u003e` element extracts the deployment ID from the request URL path. This is useful for tracking usage per model deployment.\n\n  ```\n  xml\n  \n  \n  Copy code\n  \u003cset-variable name=\"deploymentId\" value=\"@(context.Request.Url.Path.Split('/').ElementAtOrDefault(3))\" /\u003e\n  ```\n\n- **Token Limit**: The `\u003cazure-openai-token-limit\u003e` policy limits the number of tokens that can be consumed per minute per subscription.\n\n  ```\n  xmlCopy code\u003cazure-openai-token-limit\n      tokens-per-minute=\"10000000\"\n      counter-key=\"@(context.Subscription.Id)\"\n      estimate-prompt-tokens=\"true\"\n      tokens-consumed-header-name=\"consumed-tokens\"\n      remaining-tokens-header-name=\"remaining-tokens\" /\u003e\n  ```\n\n  - `tokens-per-minute`: The maximum number of tokens allowed per minute. Adjust this value according to your cost management strategy.\n  - `counter-key`: The key used to track the token count. Using `context.Subscription.Id` enforces the limit per subscription.\n  - `estimate-prompt-tokens`: When set to `true`, includes an estimate of the prompt tokens in the token count.\n  - `tokens-consumed-header-name` and `remaining-tokens-header-name`: Custom header names to include in the response, indicating tokens consumed and remaining.\n\n- **Emit Token Metrics**: The `\u003cazure-openai-emit-token-metric\u003e` policy emits metrics for token usage, which can be used for monitoring and reporting.\n\n  ```\n  xmlCopy code\u003cazure-openai-emit-token-metric\u003e\n      \u003cdimension name=\"API ID\" /\u003e\n      \u003cdimension name=\"Subscription ID\" /\u003e\n      \u003cdimension name=\"User ID\" /\u003e\n      \u003cdimension name=\"Product ID\" /\u003e\n      \u003c!-- Add the extracted deployment ID as a custom dimension --\u003e\n      \u003cdimension name=\"Deployment ID\" value=\"@(context.Variables.GetValueOrDefault\u003cstring\u003e(\"deploymentId\", \"unknown\"))\" /\u003e\n  \u003c/azure-openai-emit-token-metric\u003e\n  ```\n\n  - Each `\u003cdimension\u003e` element adds a custom dimension to the emitted metric. Including `Deployment ID` helps in tracking usage per model deployment.\n\n- **Authentication with Managed Identity**: The `\u003cauthentication-managed-identity\u003e` policy uses Managed Identity to authenticate with Azure Cognitive Services.\n\n  ```\n  xml\n  \n  \n  Copy code\n  \u003cauthentication-managed-identity resource=\"https://cognitiveservices.azure.com/\" /\u003e\n  ```\n\n#### Steps to Implement\n\n1. **Configure the Policy**: Add the above policy to your API in APIM under the inbound processing section.\n2. **Adjust Token Limits**: Modify the `tokens-per-minute` value to set the desired token limit per subscription.\n3. **Monitor Metrics**:\n   - Use Azure Monitor or Application Insights to collect and analyze the emitted metrics.\n   - Set up dashboards and alerts based on token consumption to proactively manage costs.\n4. **Communicate Limits to Clients**:\n   - Inform your API consumers about the token limits.\n   - Clients can check the `consumed-tokens` and `remaining-tokens` headers in the response to monitor their usage.\n\n#### Benefits\n\n- **Cost Control**: By limiting the number of tokens per subscription, you prevent excessive usage that could lead to unexpectedly high costs.\n- **Transparency**: Emitting token metrics with custom dimensions allows for detailed usage analysis, enabling charge-back models or internal billing.\n- **Scalability**: Implementing token limits ensures that resources are fairly distributed among consumers, improving overall system performance.\n\n#### Example Response Headers\n\nWhen clients make requests, they can examine the response headers to see their token usage:\n\n```\nyamlCopy codeconsumed-tokens: 1500\nremaining-tokens: 9850000\n```\n\n#### Handling Limit Exceeded Errors\n\nIf a client exceeds the token limit, APIM will return a **429 Too Many Requests** error. You can customize the error response using APIM policies to provide more context.\n\n```\nxmlCopy code\u003con-error\u003e\n    \u003cbase /\u003e\n    \u003cchoose\u003e\n        \u003cwhen condition=\"@(context.Response.StatusCode == 429)\"\u003e\n            \u003creturn-response\u003e\n                \u003cset-status code=\"429\" reason=\"Too Many Requests\" /\u003e\n                \u003cset-header name=\"Retry-After\" exists-action=\"override\"\u003e\n                    \u003cvalue\u003e60\u003c/value\u003e\n                \u003c/set-header\u003e\n                \u003cset-body\u003e@{\n                    return @\"{\n                        \"\"error\"\": {\n                            \"\"code\"\": \"\"TooManyTokens\"\",\n                            \"\"message\"\": \"\"Token limit exceeded. Please retry after some time.\"\"\n                        }\n                    }\";\n                }\u003c/set-body\u003e\n            \u003c/return-response\u003e\n        \u003c/when\u003e\n    \u003c/choose\u003e\n\u003c/on-error\u003e\n```\n\n#### Monitoring and Reporting\n\nBy emitting token metrics with custom dimensions, you can set up monitoring and reporting to track token consumption per subscription, deployment, and other dimensions. This can be achieved using:\n\n- **Azure Monitor Metrics**: Collect and analyze the custom metrics emitted by APIM.\n- **Log Analytics**: Aggregate logs and perform queries to generate usage reports.\n- **Alerts**: Configure alerts to notify when token usage approaches limits.\n- **Power BI**: Configure reports that connect to Log Analytics data sources.\n\n##### Log Analytics workspace via App Insights\n\n![Log Analytics report on use](./images/log-analytics-use-report.png)\n\n##### Power BI\n\n![Power BI Report on Use](./images/power-bi-use-report.png)\n\n#### Implementing Charge-back Models\n\nWith detailed metrics, you can implement charge-back models where internal teams or external customers are billed based on their actual usage. By tracking token consumption per subscription, you can allocate costs accurately.\n\n### Example: Setting Up a Charge-back Report\n\nUse this KQL Query [AzureOpenAI-with-APIM/kql_queries/KQL-Token_Tracking_and_Cost.kql at main · microsoft/AzureOpenAI-with-APIM](https://github.com/microsoft/AzureOpenAI-with-APIM/blob/main/kql_queries/KQL-Token_Tracking_and_Cost.kql)\n\n1. **Collect Metrics**: Ensure that the emitted metrics are being collected in Azure Monitor or Application Insights.\n\n2. **Create a Log Analytics Workspace**: If you haven't already, create a Log Analytics workspace to store and query your metrics.\n\n3. **Query Metrics**: Use Kusto Query Language (KQL) to query the metrics and aggregate token usage per subscription or deployment.\n\n   ```\n   customMetrics\n   | where name != \"_APPRESOURCEPREVIEW_\" // Exclude unwanted records\n   | where isnotempty(tostring(customDimensions['Deployment ID'])) // Only include records with a Deployment ID\n   | extend \n       subscriptionId = tostring(customDimensions['Subscription ID']),\n       deploymentId = tostring(customDimensions['Deployment ID']),\n       tokens = toreal(value), // Extract the token count\n       tokenType = case(\n           name == \"Prompt Tokens\", \"Prompt Tokens\",\n           name == \"Completion Tokens\", \"Completion Tokens\", \n           \"Other\") // Identify token type\n   | where tokenType in (\"Prompt Tokens\", \"Completion Tokens\") // Filter to relevant token types\n   | extend \n       // Calculate costs based on Deployment ID and Token Type, rounded to 2 decimal places\n       promptTokenCost = round(case(\n           deploymentId == \"gpt-4o\" and tokenType == \"Prompt Tokens\", tokens / 1000 * 0.03,\n           deploymentId == \"gpt-4o-global\" and tokenType == \"Prompt Tokens\", tokens / 1000 * 0.04,\n           deploymentId == \"gpt-4\" and tokenType == \"Prompt Tokens\", tokens / 1000 * 0.02,\n           deploymentId == \"gpt-35-turbo\" and tokenType == \"Prompt Tokens\", tokens / 1000 * 0.0015,\n           0.0), 3),\n       completionTokenCost = round(case(\n           deploymentId == \"gpt-4o\" and tokenType == \"Completion Tokens\", tokens / 1000 * 0.06,\n           deploymentId == \"gpt-4o-global\" and tokenType == \"Completion Tokens\", tokens / 1000 * 0.07,\n           deploymentId == \"gpt-4\" and tokenType == \"Completion Tokens\", tokens / 1000 * 0.05,\n           deploymentId == \"gpt-35-turbo\" and tokenType == \"Completion Tokens\", tokens / 1000 * 0.002,\n           0.0), 3)\n   | summarize \n       totalPromptTokens = sumif(tokens, tokenType == \"Prompt Tokens\"), \n       totalCompletionTokens = sumif(tokens, tokenType == \"Completion Tokens\"),\n       totalPromptTokenCost = round(sumif(promptTokenCost, tokenType == \"Prompt Tokens\"), 2), \n       totalCompletionTokenCost = round(sumif(completionTokenCost, tokenType == \"Completion Tokens\"), 2)\n       by subscriptionId, deploymentId // Group by Subscription ID and Deployment ID\n   | extend \n       totalCost = round(totalPromptTokenCost + totalCompletionTokenCost, 2) // Add total cost, rounded to 2 decimal places\n   | order by totalCost desc // Sort by total cost\n   ```\n\n4. **Generate Reports**: Use Azure Dashboards or Power BI to visualize the data and create reports for charge-back.\n\n5. **Automate Billing**: Export the reports or integrate with billing systems to automate the charge-back process.\n\n## Cost Forecasting\n\n- Provide cost management per subscription\n\n## Access Control\n\nTBD\n\n### Managed Identity\n\n#### APIM to Azure OpenAI\n\nTBD\n\n#### Application to APIM\n\nTBD\n\n#### Client to APIM\n\nTBD\n\n### Tokens and Keys\n\n#### APIM to Azure OpenAI (SAS Token)\n\nTBD\n\n#### Application to APIM (Subscription Key)\n\nTBD\n\n#### Client to APIM (Subscription Key)\n\nTBD\n\n## Tying it All together\n\nRead through the following steps to setup interacting with APIM and how to use consoles or .net to programatically interact with Azure OpenAI via APIM.\n\n### Get your Azure OpenAI Model Name\n\nTo determine if you have one or more models deployed, visit the AI Studio. Here you can determine if you need to create a model or use an existing model. You will use the model name when quering the Azure OpenAI API via your APIM.\n\n![Get your Azure OpenAI Model Name](./images/get-your-aoai-model-name.png)\n\n1. Navigate to your Azure OpenAI resource in Azure\n2. Select **Model deployments**\n3. Select **Manage Deployments**\n\n4. Review your models and copy the **Deployment name** of the model you want to use\n\n### Get your APIM Subscription Key\n\nThe subscription key for APIM is collected at the Subscription section of the APIM resource, regardless if you are in Azure Commercial or Government.\n\nYou can use this key for testing or as an example on how to create subscriptions to provide access to you Azure OpenAI service. Instead of sharing your Azure OpenAI Key, you create subscriptions in APIM and share this key, then you can analyze and monitor usage, provide guardrails for usage, and manage access.\n\n![Get your APIM Subscription Key](./images/get-your-apim-sub-key.png)\n\n1. Navigate to your new APIM\n2. Select **Subscriptions** from the menu\n3. Select **...**\n4. Select **Show/Hide keys**\n5. Select **copy icon**\n\n### Get your APIM Azure OpenAI URL\n\nThe URL for APIM is collected at the Overview section of the APIM resource, regardless if you are in Azure Commercial or Government.\n\n![Get your APIM Azure OpenAI URL](./images/get-your-apim-url.png)\n\n### Query Examples\n\nUsing your Azure OpenAI model, API version, APIM URL, and APIM subscription key you can now execute Azure OpenAI queries against your APIM URL instead of your Azure OpenAI URL. This means you can create new subscription keys for anyone or any team who needs access to Azure OpenAI instead of deploying new Azure OpenAI services.\n\n#### curl\n\n##### PowerShell 7\n\nCopy and paste this script into a text editor or Visual Studio code (VSC).\n\nModify by including your values, then copy and paste all of it into PowerShell 7 terminal or run from VSC.\n\n- [Link to example.ps1 file in the repo](https://github.com/microsoft/AzureOpenAI-with-APIM/blob/main/example_code/example.ps1)\n\n\u003e [!NOTE]\n\u003e\n\u003e Modify the \"CONTENT\" line for the system role and the user role to support your development and testing.\n\n```powershell\n# Update these values to match your environment\n$apimUrl = 'THE_HTTPS_URL_OF_YOUR_APIM_INSTANCE'\n$deploymentName = 'DEPLOYMENT_NAME'\n$apiVersion = '2024-02-15-preview'\n$subscriptionKey = 'YOUR_APIM_SUBSCRIPTION_KEY'\n\n# Construct the URL\n$url = \"$apimUrl/deployments/$deploymentName/chat/completions?api-version=$apiVersion\"\n\n# Headers\n$headers = @{\n    \"Content-Type\" = \"application/json\"\n    \"Ocp-Apim-Subscription-Key\" = $subscriptionKey\n}\n\n# JSON Body\n$body = @{\n    messages = @(\n        @{\n            role = \"system\"\n            content = \"You are an AI assistant that helps people find information.\"\n        },\n        @{\n            role = \"user\"\n            content = \"What are the differences between Azure Machine Learning and Azure AI services?\"\n        }\n    )\n    temperature = 0.7\n    top_p = 0.95\n    max_tokens = 800\n} | ConvertTo-Json\n\n# Invoke the API\n$response = Invoke-RestMethod -Uri $url -Method Post -Headers $headers -Body $body\n\n# Output the response\n$response.choices.message.content\n```\n\n##### Bash\n\nCopy and paste this script into a text editor or Visual Studio code.\n\nModify by including your values, then copy and paste all of it into bash terminal, run from VSC, or create a \".sh\" file to run.\n\n- [Link  to example.sh file in the repo](https://github.com/microsoft/AzureOpenAI-with-APIM/blob/main/example_code/example.sh)\n\n\u003e [!NOTE]\n\u003e\n\u003e Modify the \"CONTENT\" line for the system role and the user role to support your development and testing.\n\n\n\n```bash\n#!/bin/bash\napimUrl=\"THE_HTTPS_URL_OF_YOUR_APIM_INSTANCE\"\ndeploymentName=\"DEPLOYHMENT_NAME\" # Probaby what you named your model, but change if necessary\napiVersion=\"2024-02-15-preview\" # Change to use the latest version\nsubscriptionKey=\"YOUR_APIM_SUBSCRIPTION_KEY\"\n\nurl=\"${apimUrl}/deployments/${deploymentName}/chat/completions?api-version=${apiVersion}\"\nkey=\"Ocp-Apim-Subscription-Key: ${subscriptionKey}\"\n\n# JSON payload\njsonPayload='{\n    \"messages\": [\n        {\n            \"role\": \"system\",\n            \"content\": \"You are an AI assistant that helps people find information.\"\n        },\n        {\n            \"role\": \"user\",\n            \"content\": \"What are the differences between Azure Machine Learning and Azure AI services?\"\n        }\n    ],\n    \"temperature\": 0.7,\n    \"top_p\": 0.95,\n    \"max_tokens\": 800\n}'\n\ncurl \"${url}\" -H \"Content-Type: application/json\" -H \"${key}\" -d \"${jsonPayload}\"\n\n```\n\n#### .net\n\nYou will most likely be using Visual Studio 202x to run this and you know what you are doing.\n\n\u003e [!NOTE]\n\u003e\n\u003e Modify the \"ChatMessage\" lines for the system role and the user role to support your development and testing.\n\n```c#\n// Note: The Azure OpenAI client library for .NET is in preview.\n// Install the .NET library via NuGet: dotnet add package Azure.AI.OpenAI --version 1.0.0-beta.5 \nusing Azure;\nusing Azure.AI.OpenAI;\n\nOpenAIClient client = new OpenAIClient(\n\tnew Uri(\"https://INSERT_APIM_URL_HERE/deployments/INSERT_DEPLOYMENT_NAME_HERE/chat/completions?api-version=INSERT_API_VERSION_HERE\"),\n\tnew AzureKeyCredential(\"INSERT_APIM_SUBSCRIPTION_KEY_HERE\"));\n\n// ### If streaming is not selected\nResponse\u003cChatCompletions\u003e responseWithoutStream = await client.GetChatCompletionsAsync(\n\t\"INSERT_MODEL_NAME_HERE\",\n\tnew ChatCompletionsOptions()\n\t{\n\t\tMessages =\n\t\t{\n\t\t\tnew ChatMessage(ChatRole.System, @\"You are an AI assistant that helps people find information.\"),\n\t\t\tnew ChatMessage(ChatRole.User, @\"What are the differences between Azure Machine Learning and Azure AI services?\"),\n\t\t},\n\t\tTemperature = (float)0,\n\t\tMaxTokens = 800,\n\t\tNucleusSamplingFactor = (float)1,\n\t\tFrequencyPenalty = 0,\n\t\tPresencePenalty = 0,\n\t});\n\n// The following code shows how to get to the content from Azure OpenAI's response\nChatCompletions completions = responseWithoutStream.Value; \nChatChoice choice = completions.Choices[0];\nConsole.WriteLine(choice.Message.Content);\n```\n\n#### Python\n\nCopy and paste this script into a text editor or Visual Studio code.\n\nModify by including your values, then copy and paste all of it into bash terminal, run from VSC, or create a \".py\" file to run.\n\n- [Link  to example.py file in the repo](https://github.com/microsoft/AzureOpenAI-with-APIM/blob/main/example_code/example.py)\n\n\u003e [!NOTE]\n\u003e\n\u003e If you are running Juypter notebooks, this provides an example on using Azure OpenAI via APIM\n\n```python\n# This code is an example of how to use the OpenAI API with Azure API Management (APIM) in a Jupyter Notebook.\nimport requests\nimport json\n\n# Set the parameters\napim_url = \"apim_url\"\ndeployment_name = \"deployment_name\"\napi_version = \"2024-02-15-preview\"\nsubscription_key = \"subscription_key\"\n\n# Construct the URL and headers\nurl = f\"{apim_url}/deployments/{deployment_name}/chat/completions?api-version={api_version}\"\nheaders = {\n    \"Content-Type\": \"application/json\",\n    \"Ocp-Apim-Subscription-Key\": subscription_key\n}\n\n# Define the JSON payload\njson_payload = {\n    \"messages\": [\n        {\n            \"role\": \"system\",\n            \"content\": \"You are an AI assistant that helps people find information.\"\n        },\n        {\n            \"role\": \"user\",\n            \"content\": \"What are the differences between Azure Machine Learning and Azure AI services?\"\n        }\n    ],\n    \"temperature\": 0.7,\n    \"top_p\": 0.95,\n    \"max_tokens\": 800\n}\n\n# Make the POST request\nresponse = requests.post(url, headers=headers, json=json_payload)\n\n# Print the response text (or you can process it further as needed)\nprint(response.text)\n```\n\n## References\n\n- [Calculating Chargebacks for Business Units/Projects Utilizing a Shared Azure OpenAI Instance - Microsoft Community Hub](https://techcommunity.microsoft.com/t5/apps-on-azure-blog/calculating-chargebacks-for-business-units-projects-utilizing-a/ba-p/3909202)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrosoft%2Fazureopenai-with-apim","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmicrosoft%2Fazureopenai-with-apim","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrosoft%2Fazureopenai-with-apim/lists"}