{"id":28533485,"url":"https://github.com/epam/ai-dial-adapter-vertexai","last_synced_at":"2026-01-28T18:01:08.387Z","repository":{"id":199531054,"uuid":"693244688","full_name":"epam/ai-dial-adapter-vertexai","owner":"epam","description":"The project implements AI DIAL API for language models and embeddings from Google Vertex AI","archived":false,"fork":false,"pushed_at":"2026-01-21T09:12:53.000Z","size":1164,"stargazers_count":11,"open_issues_count":9,"forks_count":5,"subscribers_count":17,"default_branch":"development","last_synced_at":"2026-01-23T01:54:05.647Z","etag":null,"topics":["ai-dial","llm"],"latest_commit_sha":null,"homepage":"https://dialx.ai","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/epam.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-09-18T16:28:26.000Z","updated_at":"2026-01-21T09:13:13.000Z","dependencies_parsed_at":"2024-03-07T17:44:17.256Z","dependency_job_id":"93ea3148-12c4-4b98-8a5e-6f6fc6642b4c","html_url":"https://github.com/epam/ai-dial-adapter-vertexai","commit_stats":null,"previous_names":["epam/ai-dial-adapter-vertexai"],"tags_count":39,"template":false,"template_full_name":null,"purl":"pkg:github/epam/ai-dial-adapter-vertexai","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epam%2Fai-dial-adapter-vertexai","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epam%2Fai-dial-adapter-vertexai/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epam%2Fai-dial-adapter-vertexai/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epam%2Fai-dial-adapter-vertexai/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/epam","download_url":"https://codeload.github.com/epam/ai-dial-adapter-vertexai/tar.gz/refs/heads/development","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/epam%2Fai-dial-adapter-vertexai/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28848336,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-28T15:15:36.453Z","status":"ssl_error","status_checked_at":"2026-01-28T15:15:13.020Z","response_time":57,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-dial","llm"],"created_at":"2025-06-09T17:07:14.420Z","updated_at":"2026-01-28T18:01:08.367Z","avatar_url":"https://github.com/epam.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e\n  DIAL VertexAI Adapter\n\u003c/h1\u003e\n\u003cp align=\"center\"\u003e\n  \u003cp align=\"center\"\u003e\n  \u003ca href=\"https://dialx.ai/\"\u003e\n    \u003cimg src=\"https://dialx.ai/dialx_logo.svg\" alt=\"About DIALX\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\u003ch4 align=\"center\"\u003e\n  \u003ca href=\"https://discord.gg/ukzj9U9tEe\"\u003e\n    \u003cimg src=\"https://img.shields.io/static/v1?label=DIALX%20Community%20on\u0026message=Discord\u0026color=blue\u0026logo=Discord\u0026style=flat-square\" alt=\"Discord\"\u003e\n  \u003c/a\u003e\n\u003c/h4\u003e\n\n- [Overview](#overview)\n  - [Supported models](#supported-models)\n    - [Chat completion models](#chat-completion-models)\n      - [Image editing in Gemini 2.5 Flash Image](#image-editing-in-gemini-25-flash-image)\n      - [Configurable models](#configurable-models)\n        - [Imagen models](#imagen-models)\n        - [Veo models](#veo-models)\n        - [Gemini 2.5, Gemini 3 models](#gemini-25-gemini-3-models)\n        - [Gemini 2.5 Flash Image model](#gemini-25-flash-image-model)\n        - [Claude models](#claude-models)\n      - [Google Search grounding](#google-search-grounding)\n      - [Code Interpreter tool](#code-interpreter-tool)\n    - [Embedding models](#embedding-models)\n  - [Environment variables](#environment-variables)\n    - [Default `max_tokens` for Claude models](#default-max_tokens-for-claude-models)\n  - [Compatibility mode](#compatibility-mode)\n    - [Compatibility configuration in DIAL Core config](#compatibility-configuration-in-dial-core-config)\n    - [Compatibility configuration in Adapter](#compatibility-configuration-in-adapter)\n  - [Load balancing](#load-balancing)\n    - [Global endpoint](#global-endpoint)\n  - [Prompt caching](#prompt-caching)\n    - [Implicit caching](#implicit-caching)\n  - [Authentication](#authentication)\n    - [GCP Vertex AI](#gcp-vertex-ai)\n    - [Anthropic API / Google AI Platform](#anthropic-api--google-ai-platform)\n    - [Anthropic Foundry](#anthropic-foundry)\n  - [Development](#development)\n    - [Development environment](#development-environment)\n    - [IDE configuration](#ide-configuration)\n    - [Make on Windows](#make-on-windows)\n    - [Run](#run)\n    - [Lint](#lint)\n    - [Test](#test)\n    - [Clean](#clean)\n\n---\n\n# Overview\n\nLLM Adapters unify the APIs of respective LLMs to align with the Unified Protocol of DIAL Core. Each Adapter operates within a dedicated container. Multi-modality allows supporting non-textual communications such as image-to-text, text-to-image, file transfers and more.\n\nThe project implements [AI DIAL API](https://dialx.ai/dial_api) for language models and embedding models from [Vertex AI](https://console.cloud.google.com/vertex-ai).\n\n![ai-dial-core](https://docs.dialx.ai/assets/images/adapters-62587fb74cfb1c4225c20c08273ec5bc.svg)\n\n---\n\n## Supported models\n\n### Chat completion models\n\nThe following models support `POST $SERVER_ORIGIN/openai/deployments/$MODEL_ID/chat/completions` endpoint along with an optional support of the feature endpoints:\n\n- `POST $SERVER_ORIGIN/openai/deployments/$MODEL_ID/tokenize`\n- `POST $SERVER_ORIGIN/openai/deployments/$MODEL_ID/truncate_prompt`\n- `POST $SERVER_ORIGIN/openai/deployments/$MODEL_ID/configuration`\n\n|Model|Model ID|Modality|`/tokenize`|`/truncate_prompt`|tools/functions support|`/configuration`|\n|---|---|---|---|---|---|---|\n|Gemini 3 Pro|gemini-3-pro-preview|(text/pdf/image/audio/video)-to-text|✅|✅|✅|✅|\n|Gemini 3 Flash|gemini-3-flash-preview|(text/pdf/image/audio/video)-to-text|✅|✅|✅|✅|\n|Gemini 3 Pro Image|gemini-3-pro-image-preview|(text/image)-to-(text/image)|✅|✅|✅|✅|\n|Gemini 2.5 Flash|gemini-2.5-flash|(text/pdf/image/audio/video)-to-text|✅|✅|✅|✅|\n|Gemini 2.5 Flash Image|gemini-2.5-flash-image|(text/image)-to-(text/image)|✅|✅|✅|✅|\n|Gemini 2.5 Pro|gemini-2.5-pro|(text/pdf/image/audio/video)-to-text|✅|✅|✅|✅|\n|Gemini 2.0 Flash Lite|gemini-2.0-flash-lite-001|(text/pdf/image/audio/video)-to-text|✅|✅|✅|❌|\n|Gemini 2.0 Flash|gemini-2.0-flash-exp|(text/pdf/image/audio/video)-to-(text/image)|✅|✅|✅|❌|\n|Gemini 2.0 Flash|gemini-2.0-flash-001|(text/pdf/image/audio/video)-to-text|✅|✅|✅|❌|\n|Claude 4.5 Sonnet|claude-sonnet-4-5@20250929|(pdf/text/image)-to-text|✅|✅|✅|✅|\n|Claude 4.5 Haiku|claude-haiku-4-5@20251001|(pdf/text/image)-to-text|✅|✅|✅|✅|\n|Claude 4.1 Opus|claude-opus-4-1@20250805|(pdf/text/image)-to-text|✅|✅|✅|✅|\n|Claude 4 Opus|claude-opus-4@20250514|(pdf/text/image)-to-text|✅|✅|✅|✅|\n|Claude 4 Sonnet|claude-sonnet-4@20250514|(pdf/text/image)-to-text|✅|✅|✅|✅|\n|Claude 3.7 Sonnet|claude-3-7-sonnet@20250219|(pdf/text/image)-to-text|✅|✅|✅|✅|\n|Claude 3 Opus|claude-3-opus@20240229|(text/image)-to-text|✅|✅|✅|✅|\n|Claude 3.5 Sonnet v2|claude-3-5-sonnet-v2@20241022|(pdf/text/image)-to-text|✅|✅|✅|✅|\n|Claude 3.5 Sonnet|claude-3-5-sonnet@20240620|(pdf/text/image)-to-text|✅|✅|✅|✅|\n|Claude 3.5 Haiku|claude-3-5-haiku@20241022|(pdf/text)-to-text|✅|✅|✅|✅|\n|Claude 3 Haiku|claude-3-haiku@20240307|(text/image)-to-text|✅|✅|✅|✅|\n|Imagen 4.0|imagen-4.0-(generate-preview-06-06\\|fast-generate-preview-06-06\\|ultra-generate-preview-06-06\\|generate-001\\|fast-generate-001\\|ultra-generate-001)|text-to-image|✅|✅|❌|✅|\n|Imagen 3.0|imagen-3.0-(generate-001\\|generate-002\\|fast-generate-001)|text-to-image|✅|✅|❌|✅|\n|Imagen 2|imagegeneration@005|text-to-image|✅|✅|❌|✅|\n|Veo 3.1 Fast Generate|veo-3.1-fast-generate-(001\\|preview)|text-to-video|✅|✅|❌|✅|\n|Veo 3.1 Generate|veo-3.1-generate-(001\\|preview)|text-to-video|✅|✅|❌|✅|\n|Veo 3.0 Fast Generate|veo-3.0-fast-generate-(001\\|preview)|text-to-video|✅|✅|❌|✅|\n|Veo 3.0 Generate|veo-3.0-generate-(001\\|preview)|text-to-video|✅|✅|❌|✅|\n\nThe models that support `/truncate_prompt` do also support `max_prompt_tokens` chat completion request parameter.\n\n#### Image editing in Gemini 2.5 Flash Image\n\nGemini 2.5 Flash Image and Gemini 3 Pro Image models support both image generation and image editing.\nThis enables the following use case:\n\n```txt\nuser: generate an image of a cat sitting on a sofa\nassistant: \u003cimage attachment #1\u003e\nuser: replace the cat with a dog\nassistant: \u003cimage attachment #2\u003e\n```\n\nThis scenario works out of the box with API integrations.\n\nHowever, it wouldn't work for interactions via [DIAL Chat](https://github.com/epam/ai-dial-chat/tree/0.40.0), since it removes all attachments from the assistant messages by default. To change this default behavior, set the `assistantAttachmentsInRequestSupported` flag to `true` in the [DIAL Core configuration](https://github.com/epam/ai-dial-core/blob/0.37.0/docs/dynamic-settings/models.md#modelsmodel_namefeatures) for the Gemini deployment in question:\n\n```json\n{\n  \"models\": {\n    \"dial-gemini-deployment-id\": {\n      \"type\": \"chat\",\n      \"endpoint\": \"${VERTEXAI_ADAPTER_ORIGIN}/openai/deployments/gemini-2.5-flash-image/chat/completions\",\n      \"features\": {\n        \"assistantAttachmentsInRequestSupported\": true\n      }\n    }\n  }\n}\n```\n\n#### Configurable models\n\nCertain models support configuration via the `/configuration` endpoint.\nGET request to this endpoint returns the schema of the model configuration in [JSON Schema](https://json-schema.org/) format.\nSuch models expect that `custom_fields.configuration` field of the `chat/completions` request will contain a JSON value that conforms to the schema.\nThe `custom_fields.configuration` field is optional iff each field in the schema is optional too.\n\n##### Imagen models\n\nThe Imagen models support configuration of parameters specific for image-generation such as negative prompt, aspect ratio and watermarking. See the complete list of configurable parameters at the [Imagen API documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/imagen-api#python_1).\n\n```json\n{\n  \"messages\": [{\"role\": \"user\", \"content\": \"forest meadow\"}],\n  \"custom_fields\": {\n    \"configuration\": {\n      \"add_watermark\": false,\n      \"negative_prompt\": \"trees\",\n      \"aspect_ratio\": \"16:9\"\n    }\n  }\n}\n```\n\n##### Veo models\n\nThe Veo models support configuration of parameters specific for video-generation such as aspect ratio, compression quality and duration seconds. See the complete list of configurable parameters at the [Veo API documentation](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/model-reference/veo-video-generation#parameters).\n\n```json\n{\n  \"messages\": [{\"role\": \"user\", \"content\": \"forest meadow\"}],\n  \"custom_fields\": {\n    \"configuration\": {\n      \"aspect_ratio\": \"16:9\",\n      \"compression_quality\": \"optimized\",\n      \"duration_seconds\": 4\n    }\n  }\n}\n```\n\n##### Gemini 2.5, Gemini 3 models\n\nThe Gemini 2.5 and Gemini 3 series models support configuration of the [thinking parameters](https://ai.google.dev/gemini-api/docs/thinking):\n\n```json\n{\n  \"custom_fields\": {\n    \"configuration\": {\n      \"thinking\": {\n        \"include_thoughts\": true,\n        \"thinking_budget\": 2048\n      }\n    }\n  }\n}\n```\n\nThe thought summaries are printed into a dedicated `Thinking` stage given that `include_thoughts` is set to `true`.\n\nThe token budget for thinking may be set to be unlimited via `thinking_budget=-1`.\n\nThe Gemini 3 series model also supports [thinking_level](https://ai.google.dev/gemini-api/docs/gemini-3?thinking=low#thinking_level) parameter that could be set via the [reasoning_effort](https://platform.openai.com/docs/api-reference/chat/create#chat_create-reasoning_effort) field from the OpenAI API:\n\n```json\n{\n  \"messages\": [{\"role\": \"user\", \"content\": \"Explain quantum computing in simple terms.\"}],\n  \"reasoning_effort\": \"none|minimum|low|medium|high\"\n}\n```\n\n\u003e [!NOTE]\n\u003e You cannot use both `reasoning_effort` and the `thinking_budget` parameters in the same request.\n\n##### Gemini 2.5 Flash Image model\n\nThe Gemini 2.5 Flash Image support configuration of parameters controlling generation of images:\n\n```json\n{\n  \"custom_fields\": {\n    \"configuration\": {\n      \"image_config\": {\n        \"aspect_ratio\": \"21:9\",\n        \"image_size\": \"4K\"\n      }\n    }\n  }\n}\n```\n\nConsult the [documentation](https://ai.google.dev/gemini-api/docs/image-generation#aspect_ratios_and_image_size) for the possible values of these parameters and their defaults.\n\n##### Claude models\n\nThe Claude models accept a configuration flag that enables document citations in the generated output. The flag is false by default.\n\n```json\n{\n  \"custom_fields\": {\n    \"configuration\": {\n      \"enable_citations\": true\n    }\n  }\n}\n```\n\nNot every Claude model supports citations. Refer to the [official documentation](https://docs.anthropic.com/en/docs/build-with-claude/citations) before utilizing any flags.\n\nBesides that Claude models support beta flags.\nThe whole list of flags could be found in the [Anthropic SDK](https://github.com/anthropics/anthropic-sdk-python/blob/main/src/anthropic/types/anthropic_beta_param.py).\n\nThe most notable beta flags are:\n\n|Configuration|Comment|Scope|\n|---|---|---|\n|`{\"betas\": [\"token-efficient-tools-2025-02-19\"]}`|[Token-efficient tool use](https://docs.anthropic.com/en/docs/build-with-claude/tool-use/token-efficient-tool-use)|Claude 3.7 Sonnet|\n|`{\"betas\": [\"output-128k-2025-02-19\"]}`|[Extended output length](https://docs.anthropic.com/en/docs/build-with-claude/extended-thinking#extended-output-capabilities-beta)|Claude 3.7 Sonnet|\n\nNot every model supports all flags. Refer to the official documentation before utilizing any flags.\n\n#### Google Search grounding\n\nGemini models support [Grounding with Google Search](https://ai.google.dev/gemini-api/docs/google-search?lang=python#google-search-retrieval). It's enabled by the `google_search` static tool:\n\n```json\n{\n  \"message\": [\n    {\n      \"user\": \"role\",\n      \"content\": \"What are the latest GenAI news?\"\n    }\n  ],\n  \"tools\": [\n    {\n      \"type\": \"static_function\",\n      \"static_function\": {\n        \"name\": \"google_search\"\n      }\n    }\n  ]\n}\n```\n\nThe response will include DIAL attachments with citations from relevant URLs fetched by Google Search.\n\nRefer to the official [documentation](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/model-reference/grounding#supported_models) for the list of models supporting Google Search grounding.\n\n#### Code Interpreter tool\n\nGemini models support [Code Interpreter tool](https://ai.google.dev/gemini-api/docs/code-execution?lang=python). It's enabled by the `code_execution` static tool:\n\n```json\n{\n  \"message\": [\n    {\n      \"user\": \"role\",\n      \"content\": \"What is the sum of the first 50 prime numbers? Generate and run code for the calculation, and make sure you get all 50.\"\n    }\n  ],\n  \"tools\": [\n    {\n      \"type\": \"static_function\",\n      \"static_function\": {\n        \"name\": \"code_execution\"\n      }\n    }\n  ]\n}\n```\n\nThe response will include stage titled `Code execution` for the code generated by the model and the result of its execution.\n\n---\n\n### Embedding models\n\nThe following models support `$SERVER_ORIGIN/openai/deployments/$MODEL_ID/embeddings` endpoint:\n\n|Model|Model ID|Language support|Modality|\n|---|---|---|---|\n|Gemini Embeddings|gemini-embedding-001|Multilingual|text-to-embedding|\n|Gecko Embeddings for Text V1|textembedding-gecko@001|English|text-to-embedding|\n|Gecko Embeddings for Text V3|textembedding-gecko@003|English|text-to-embedding|\n|Embeddings for Text|text-embedding-(004\\|005)|English|text-to-embedding|\n|Gecko Embeddings for Text Multilingual|textembedding-gecko-multilingual@001|Multilingual|text-to-embedding|\n|Embeddings for Text Multilingual|text-multilingual-embedding-002|Multilingual|text-to-embedding|\n|Multimodal embeddings|multimodalembedding@001|English|(text/image)-to-embedding|\n\n---\n\n## Environment variables\n\nCopy `.env.example` to `.env` and customize it for your environment:\n\n|Variable|Default|Description|\n|---|---|---|\n|GOOGLE_APPLICATION_CREDENTIALS||Filepath to JSON with [credentials](https://cloud.google.com/docs/authentication/application-default-credentials#GAC)|\n|DEFAULT_REGION||Default region for Vertex AI (e.g. \"us-central1\")|\n|GCP_PROJECT_ID||GCP project ID|\n|LOG_LEVEL|INFO|Log level. Use DEBUG for dev purposes and INFO in prod|\n|AIDIAL_LOG_LEVEL|WARNING|AI DIAL SDK log level|\n|WEB_CONCURRENCY|1|Number of workers for the server|\n|DIAL_URL||URL of the core DIAL server. Optional. Used to access images stored in the DIAL File storage|\n|COMPATIBILITY_MAPPING|{}|**Deprecated** in favour of [compatibility configuration in DIAL Core config](#compatibility-configuration-in-dial-core-config). A JSON dictionary that maps VertexAI deployments that **aren't supported** by the Adapter to the VertexAI deployments that **are supported** by the Adapter _(see the [Supported models](#supported-models)_ section). Find more details in the [compatibility mode](#compatibility-configuration-in-adapter) section.|\n|CLAUDE_DEFAULT_MAX_TOKENS|1536|The default value of `max_tokens` chat completion parameter if it is not provided in the request.\u003cbr\u003e**:warning: Using the variable is discouraged**.\u003cbr\u003eConsider configuring the default in the DIAL Core Config instead as demonstrated in the [example below](#default-max_tokens-for-claude-models).|\n|GOOGLE_GENAI_MAX_RETRY_ATTEMPTS|0|How many times to retry Google GenAI chat model requests when the provider returns a retriable error|\n|ANTHROPIC_MAX_RETRY_ATTEMPTS|0|How many times to retry Anthropic chat model requests when the provider returns a retriable error|\n\n### Default `max_tokens` for Claude models\n\nUnlike Gemini models, Claude models require the `max_tokens` parameter in the chat completion request.\n\nWe recommend configuring `max_tokens` default value on a per-model basis in the DIAL Core Config, for example:\n\n```json\n{\n    \"models\": {\n        \"dial-claude-deployment-id\": {\n            \"type\": \"chat\",\n            \"description\": \"...\",\n            \"endpoint\": \"...\",\n            \"defaults\": {\n                \"max_tokens\": 2048\n            }\n        }\n    }\n}\n```\n\nIf the default is missing in the DIAL Core Config, it will be taken from the `CLAUDE_DEFAULT_MAX_TOKENS` environment variable.\nHowever, we strongly recommend not to rely on this variable and instead configure the defaults in the DIAL Core Config.\nSuch a **per-model** configuration is operationally cleaner since all the information relevant to tokens _(like pricing and token limits)_ is kept in the same place.\n\nThe default value set in the DIAL Core Config takes precedence over the one configured in the adapter.\n\nMake sure the default doesn't exceed Claude's [max output tokens](https://docs.anthropic.com/en/docs/about-claude/models/all-models#model-comparison-table), otherwise, you will receive an error like this one: `max_tokens: 10000 \u003e 8192, which is the maximum allowed number of output tokens for claude-3...)`.\n\n## Compatibility mode\n\nThe Adapter supports a predefined list of VertexAI deployments. The [Supported models](#supported-models) section lists the models. These models could be accessed via `/openai/deployments/$MODEL_ID/(chat_completions|embeddings)` endpoints. The Adapter won't recognize any other deployment name and will result in `404` error.\n\nNow, suppose VertexAI has just released a new version of a model, e.g. `gemini-2.0-flash-006` that is a better version of an older `gemini-2.0-flash-001` model.\n\nImmediately after the release, the former model is **unsupported** by the Adapter, but the latter is **supported**.\nTherefore, the request to `openai/deployments/gemini-2.0-flash-006/chat/completions` will result in 404 error.\n\nIt will take some time for the Adapter to catch up with VertexAI - support the v6 model and publish the release with the fix.\n\nWhat to do in the meantime? Presumably, the v6 model is backward compatible with v1, so we may try to run v6 in **the compatibility mode** - that is to convince the Adapter to process v6 request as if it's v1 request with the only difference that the final upstream request to GCP VertexAI will be to v6 and not v1.\n\nThere are two way to enable compatibility mode in the adapter.\n\n### Compatibility configuration in DIAL Core config\n\nIt's possible to define compatible model on per-upstream basis in the DIAL Core configuration.\n\nE.g. the following configuration enables `gemini-2.0-flash-006` model _(a hypothetical model that isn't supported by the Adapter natively)_ via `gemini-2.0-flash-001` model _(that is supported by the Adapter natively)_:\n\n```json\n{\n  \"models\": {\n    \"dial-deployment-id-for-claude-3-5\": {\n      \"type\": \"chat\",\n      \"endpoint\": \"${ADAPTER_ORIGIN}/deployments/gemini-2.0-flash-006/chat/completions\",\n      \"upstreams\": [\n        {\n          \"extraData\": {\n            \"compatible_model_id\": \"gemini-2.0-flash-001\"\n          }\n        }\n      ]\n    }\n  }\n}\n```\n\nThe given configuration enables the adapter to handle requests to the `gemini-2.0-flash-006` deployment.\nThe requests will be processed by the same pipeline as `gemini-2.0-flash-001`, but the call to GCP VertexAI will be done to `gemini-2.0-flash-006` deployment name.\n\nNaturally, this will only work if the APIs of v1 and v6 deployments are compatible:\n\n1. The requests utilizing the modalities supported by both v1 and v6 will work just fine.\n2. However, the requests with modalities that are supported by v6 and aren't supported by v1, won't be processed correctly. You will have to wait until the adapter supports the v6 deployment natively.\n\nWhen a version of the adapter supporting the v6 model is released, you may migrate to it and safely remove the `compatible_model_id` from the DIAL Core config.\n\nNote that setting `compatible_model_id=imagen-4.0-generate-001` will be ineffectual, since the APIs of the two model and their capabilities are drastically different.\n\n\u003e [!IMPORTANT]\n\u003e If the DIAL deployment has many upstreams, the `compatible_model_id` field should be set in all of the upstreams.\n\n### Compatibility configuration in Adapter\n\n\u003e [!IMPORTANT]\n\u003e Model compatibility configuration via `COMPATIBILITY_MAPPING` env variable has been deprecated in favour of [configuration on the DIAL Core side](#compatibility-configuration-in-dial-core-config) in DIAL Core config.\n\n`COMPATIBILITY_MAPPING` env variable enables compatibility mode on the adapter level.\nIt hold a mapping from unsupported deployment ids to supported deployment ids.\n\nE.g. the following mapping enables `gemini-2.0-flash-006` via `gemini-2.0-flash-001`:\n\n```ini\nCOMPATIBILITY_MAPPING={\"gemini-2.0-flash-006\": \"gemini-2.0-flash-001\"}\n```\n\n## Load balancing\n\nIf you use DIAL Core load balancing mechanism, you can provide `extraData` upstream setting the region and the project to use for a particular upstream:\n\n```json\n{\n  \"upstreams\": [\n    {\n      \"extraData\": {\n        \"project\": \"project1\",\n        \"region\": \"us-central1\"\n      }\n    },\n    {\n      \"extraData\": {\n        \"project\": \"project1\",\n        \"region\": \"us-east5\"\n      }\n    },\n    {\n      \"extraData\": {\n        \"project\": \"project2\"\n      }\n    },\n    {\n      \"key\": \"api-key\"\n    }\n  ]\n}\n```\n\nThe fields in the extra data override the corresponding environment variables:\n\n|`extraData` field|Env variable|\n|---|---|\n|`region`|`DEFAULT_REGION`|\n|`project`|`GCP_PROJECT_ID`|\n\n\u003e [!NOTE]\n\u003e The region and project configuration is only supported for Gemini\u003e=2 and Anthropic models.\n\n### Global endpoint\n\nUse the `global` region to enable the [global endpoint](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/locations#global-endpoint):\n\n```json\n{\n  \"upstreams\": [\n    {\n      \"extraData\": {\n        \"region\": \"global\"\n      }\n    }\n  ]\n}\n```\n\n\u003e [!NOTE]\n\u003e The global endpoint is supported only for [certain models](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/locations#supported_models) and has a few other [limitations](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/locations#limitations).\n\n## Prompt caching\n\n### Implicit caching\n\nGemini 2.5 models support [implicit context caching](https://ai.google.dev/gemini-api/docs/caching?lang=python#implicit-caching).\n\nAny request over a certain amount of tokens will be automatically cached. The token threshold triggering caching for Gemini 2.5 Flash is 1024 and for Gemini 2.5 Pro - 4096.\n\nSet `autoCachingSupported` flag in the DIAL Core config for a deployment of interest to enable this feature:\n\n```json\n{\n  \"models\": {\n    \"my-dial-gemini-deployment\": {\n      \"type\": \"chat\",\n      \"displayName\": \"Gemini 2.5 Flash\",\n      \"endpoint\": \"${VERTEXAI_ADAPTER_ORIGIN}/openai/deployments/gemini-2.5-flash/chat/completions\",\n      \"upstreams\": [\n        {\n          \"extraData\": {\n            \"region\": \"us-central1\"\n          }\n        },\n        {\n          \"extraData\": {\n            \"region\": \"us-east5\"\n          }\n        }\n      ],\n      \"features\": {\n        \"autoCachingSupported\": true\n      }\n    }\n  }\n}\n```\n\nOn a cache hit, `usage.prompt_tokens_details.cached_tokens` field reports the number of cached prompt tokens.\n\n## Authentication\n\n### GCP Vertex AI\n\nAccess to GCP Vertex AI is authenticated via Application Default Credentials ([ADC](https://cloud.google.com/docs/authentication/application-default-credentials)) with region and project configured either:\n\n1. globally via `DEFAULT_REGION` and `GCP_PROJECT_ID` environment vars, or\n2. on a [per upstream basis](#load-balancing) via `upstreams.extraData` fields in DIAL Core Config.\n\n### Anthropic API / Google AI Platform\n\nGemini\u003e=2 and Anthropic deployments could be accessed via API key. The API keys should be configured per-upstream in the DIAL Core config:\n\n```json\n{\n  \"models\": {\n    \"gemini-dial-deployment-id\": {\n      \"endpoint\": \"${ADAPTER_ORIGIN}/deployments/gemini-2.0-flash-lite-001/chat/completions\",\n      \"upstreams\": [\n        {\n          \"key\": \"gemini-api-key\"\n        }\n      ]\n    },\n    \"claude-dial-deployment-id\": {\n      \"endpoint\": \"${ADAPTER_ORIGIN}/deployments/claude-3-5-sonnet-20241022/chat/completions\",\n      \"upstreams\": [\n        {\n          \"key\": \"anthropic-api-key\",\n          \"extraData\": {\n            \"compatible_model_id\": \"claude-3-5-sonnet-v2@20241022\"\n          }\n        }\n      ]\n    }\n  }\n}\n```\n\nKeep in mind that the same Anthropic models have [different identifiers](https://docs.anthropic.com/en/docs/about-claude/models/overview#model-names) in Anthropic API and GPC Vertex AI.\n\nE.g. `claude-3-5-sonnet-v2@20241022` in GCP Vertex AI corresponds to `claude-3-5-sonnet-20241022` in Anthropic API.\n\nThe VertexAI adapter uses model names from **GCP Vertex AI**. Therefore, in order to use **Anthropic API** model name you need to specify the corresponding name from **GCP Vertex AI** in the [compatible_model_id](#compatibility-configuration-in-dial-core-config) field. Otherwise, the adapter returns 404.\n\n### Anthropic Foundry\n\nThe adapter supports access to Claude models from [Azure](https://learn.microsoft.com/en-us/azure/ai-foundry/foundry-models/how-to/use-foundry-models-claude?view=foundry\u0026preserve-view=true\u0026tabs=python) [Foundry](https://platform.claude.com/docs/en/build-with-claude/claude-in-microsoft-foundry) service.\n\n```json\n{\n  \"models\": {\n    \"claude-dial-deployment-id\": {\n      \"endpoint\": \"${VERTEXAI_ADAPTER_ORIGIN}/openai/deployments/claude-sonnet-4-520250929/chat/completions\",\n      \"upstreams\": [\n        {\n          \"endpoint\": \"https://${AZURE_FOUNDRY_RESOURCE_NAME1}.services.ai.azure.com/anthropic/v1/messages\",\n          \"key\": \"optional-azure-foundry-api-key1\"\n        },\n        {\n          \"endpoint\": \"https://${AZURE_FOUNDRY_RESOURCE_NAME2}.services.ai.azure.com/anthropic/v1/messages\",\n          \"key\": \"optional-azure-foundry-api-key2\"\n        }\n      ]\n    }\n  }\n}\n```\n\nThe [DefaultAzureCredential](https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python) is used to authenticate requests to Azure when an API key is not provided in the upstream configuration.\n\nSince the models names in Azure Foundry are different from the ones in GCP Vertex AI, you need to map them onto supported [deployment names](#supported-models) using the [compatibility mapping](#compatibility-mode):\n\n```txt\nCOMPATIBILITY_MAPPING={\"claude-sonnet-4-520250929\":\"claude-sonnet-4-5@20250929\"}\n```\n\n---\n\n## Development\n\n### Development environment\n\nThis project uses [Python\u003e=3.11](https://www.python.org/downloads/) and [Poetry\u003e=2.1.1](https://python-poetry.org/) as a dependency manager.\n\nCheck out Poetry's [documentation on how to install it](https://python-poetry.org/docs/#installation) on your system before proceeding.\n\nTo install requirements:\n\n```sh\npoetry install\n```\n\nThis will install all requirements for running the package, linting, formatting and tests.\n\n### IDE configuration\n\nThe recommended IDE is [VS Code](https://code.visualstudio.com/).\nOpen the project in VS Code and install the recommended extensions.\nVS Code is configured to use PEP-8 compatible formatter [Black](https://black.readthedocs.io/en/stable/index.html).\n\nAlternatively you can use [PyCharm](https://www.jetbrains.com/pycharm/).\nSet up the Black in PyCharm [manually](https://black.readthedocs.io/en/stable/integrations/editors.html#pycharm-intellij-idea) or\ninstall PyCharm\u003e=2023.2 with [built-in Black support](https://blog.jetbrains.com/pycharm/2023/07/2023-2/#black).\n\n### Make on Windows\n\nAs of now, Windows distributions do not include the make tool. To run make commands, the tool can be installed using\nthe following command (since [Windows 10](https://learn.microsoft.com/en-us/windows/package-manager/winget/)):\n\n```sh\nwinget install GnuWin32.Make\n```\n\nFor convenience, the tool folder can be added to the PATH environment variable as `C:\\Program Files (x86)\\GnuWin32\\bin`.\nThe command definitions inside Makefile should be cross-platform to keep the development environment setup simple.\n\n### Run\n\nRun the development server locally:\n\n```sh\nmake serve\n```\n\nRun the server from a Docker container:\n\n```sh\nmake docker_serve\n```\n\n### Lint\n\nDon't forget to run the linting before committing:\n\n```sh\nmake lint\n```\n\nTo auto-fix formatting issues run:\n\n```sh\nmake format\n```\n\n### Test\n\nTo run the unit tests locally:\n\n```sh\nmake test\n```\n\nTo run the unit tests from the Docker container:\n\n```sh\nmake docker_test\n```\n\nTo run the integration tests locally:\n\n```sh\nmake integration_tests\n```\n\n### Clean\n\nTo remove the virtual environment and build artifacts:\n\n```sh\nmake clean\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fepam%2Fai-dial-adapter-vertexai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fepam%2Fai-dial-adapter-vertexai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fepam%2Fai-dial-adapter-vertexai/lists"}