{"id":23933007,"url":"https://github.com/awslabs/generative-ai-toolkit","last_synced_at":"2025-09-11T15:33:25.659Z","repository":{"id":288548259,"uuid":"878156661","full_name":"awslabs/generative-ai-toolkit","owner":"awslabs","description":"The Generative AI Toolkit is a lightweight library for building, testing and evaluating AI agents in Python, using any of the LLMs supported by the Amazon Bedrock Converse API.","archived":false,"fork":false,"pushed_at":"2025-09-03T07:54:50.000Z","size":7536,"stargazers_count":25,"open_issues_count":1,"forks_count":9,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-09-11T07:49:08.178Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/awslabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE","maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-10-24T21:26:47.000Z","updated_at":"2025-09-04T08:33:16.000Z","dependencies_parsed_at":"2025-06-24T09:33:37.190Z","dependency_job_id":"676cbe68-1547-4b90-955c-4b6e74275b11","html_url":"https://github.com/awslabs/generative-ai-toolkit","commit_stats":null,"previous_names":["awslabs/generative-ai-toolkit"],"tags_count":22,"template":false,"template_full_name":null,"purl":"pkg:github/awslabs/generative-ai-toolkit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awslabs%2Fgenerative-ai-toolkit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awslabs%2Fgenerative-ai-toolkit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awslabs%2Fgenerative-ai-toolkit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awslabs%2Fgenerative-ai-toolkit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/awslabs","download_url":"https://codeload.github.com/awslabs/generative-ai-toolkit/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/awslabs%2Fgenerative-ai-toolkit/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274660279,"owners_count":25326172,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-11T02:00:13.660Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-06T00:29:29.122Z","updated_at":"2025-09-11T15:33:25.643Z","avatar_url":"https://github.com/awslabs.png","language":"Python","funding_links":[],"categories":["Building"],"sub_categories":["Deployment"],"readme":"# Generative AI Toolkit\n\nThe **Generative AI Toolkit** is a lightweight library for building, testing and evaluating AI agents in Python, using any of the LLMs supported by the Amazon Bedrock Converse API.\n\nCompared to other libraries out there, the Generative AI Toolkit indexes heavily on production observability, tracing, testing and evaluation, and simplicity of deployment on AWS. A typical production-grade deployment uses just AWS Lambda (or ECS, EKS), Amazon DynamoDB and Amazon CloudWatch.\n\n**Sample usage**\n\n```python\nfrom generative_ai_toolkit.agent import BedrockConverseAgent\nfrom generative_ai_toolkit.context import AgentContext\nfrom generative_ai_toolkit.test import Case, Expect\n\nagent = BedrockConverseAgent(\n model_id=\"eu.amazon.nova-micro-v1:0\",\n system_prompt=\"You are a helpful assistant. Use your tools to help the user.\",\n)\n\n\ndef weather_report(city_name: str) -\u003e str:\n \"\"\"\n Gets the current weather report for a given city\n\n Parameters\n ------\n city_name: string\n The name of the city\n \"\"\"\n\n # Example of how to add tracing to your tool implementations\n tracer = AgentContext.current().tracer\n\n with tracer.trace(\"inside-weather-report\") as span:\n span.add_attribute(\n \"attribute_name\", {\"Attribute values\": [\"can be any Python object\"]}\n )\n\n # Tool response\n return \"Sunny\"\n\n\nagent.register_tool(weather_report)\n\n# Send a message to the agent and have it stream its response back:\nfor chunk in agent.converse_stream(\"What's the weather like right now in Amsterdam?\"):\n print(chunk, end=\"\")\n\n# Assert that the weather_report tool was used (raises an error if not):\nExpect(agent.traces).tool_invocations.to_include(\"weather_report\")\n\n# Similar, but using test case with 2 turns:\ntest_case = Case(user_inputs=[\"What's the weather like right now?\", \"In Amsterdam\"])\ntraces = test_case.run(agent)\nExpect(traces).tool_invocations.to_include(\"weather_report\").with_output(\"Sunny\")\n\n# Start a new conversation, with empty history:\nagent.reset()\n\n# Stream traces instead of text chunks:\n# (Note: this also yields relevant snapshots of traces that are still underway)\nfor trace in agent.converse_stream(\n \"What's the weather like right now in Amsterdam?\", stream=\"traces\"\n):\n print(trace.as_human_readable())\n\n# Trace attributes are OpenTelemetry compatible but preserve fidelity:\ntool_trace = next(\n trace for trace in agent.traces if trace.span_name == \"inside-weather-report\"\n)\nassert tool_trace.attributes[\"attribute_name\"] == {\n \"Attribute values\": [\"can be any Python object\"]\n}\n\n# You can also collect metrics, run tests in parallel, compare different model ids\n# and other agent parameters against each other, view metrics and traces in a UI,\n# add your own tracers, use an LLM mock, expose the agent over HTTP, and much more.\n...\n```\n\n**Major features**\n\n- **Traces** are front and center in the Generative AI Toolkit––everything your agent does can be traced. Traces for the current conversation can be accessed with `agent.traces`, which returns traces from subagents too (recursively). This simplifies testing and visualization of your single or multi-agent architectures. Out-of-the-box tracers for DynamoDB and AWS X-Ray are included, and it's easy to add custom tracers. In automated tests you can use the `Expect()` class to express your test assertions against the traces.\n- For **evaluation**, use the out-of-the-box metrics such as cost, latency, cosine similarity, conciseness, or add your own custom metrics. The metrics you use while developing your agent, can continuously be run against your deployed agent in production too (asynchronously). If your agent's performance degrades, you will know! You have the full power of Amazon CloudWatch Metrics available to you to define alarms and thresholds, and catch anomalies.\n- Helpers for **mocking** the Amazon Bedrock Converse API, to create unit tests and partially mocked integration tests that are deterministic and fast. The mock supports multi-turn conversation, and dynamic response generation. With the mock you can develop and run your agent locally without needing access to an actual LLM\n- Integrates with **[Amazon Bedrock AgentCore](https://aws.amazon.com/bedrock/agentcore/)**\n- The testing and evaluation capabilities can be used for agents and LLM-based applications created with **other libraries** (for example [Strands](https://strandsagents.com/latest/documentation/docs/)).\n\nInterested? Please read on. And check out our research paper: [GENERATIVE AI TOOLKIT- INCREASING THE QUALITY OF LLM-BASED APPLICATIONS OVER THEIR WHOLE LIFE CYCLE](https://arxiv.org/abs/2412.14215).\n\n## Screenshots\n\nYou can view the traces, as well as the collected metrics, in various developer friendly ways, e.g. with the web UI:\n\n\u003cimg src=\"./assets/images/ui-conversation.png\" alt=\"UI Conversation Display Screenshot\" title=\"UI Conversation Display\" width=\"1200\"/\u003e\n\nTraces are also visible in AWS X-Ray (or any other OpenTelemetry compatible tool of your choosing):\n\n\u003cimg src=\"./assets/images/x-ray-trace-map.png\" alt=\"AWS X-Ray Trace Map Screenshot\" title=\"AWS X-Ray Trace Map\" width=\"1200\"/\u003e\n\nDetails of the traces, such as the LLM inputs and outputs, are visible in the trace timelines as metadata:\n\n\u003cimg src=\"./assets/images/x-ray-trace-segments-timeline.png\" alt=\"AWS X-Ray Trace Segments Timeline Screenshot\" title=\"AWS X-Ray Trace Segments Timeline\" width=\"1200\"/\u003e\n\nMetrics can be emitted to Amazon CloudWatch easily, so you can create dashboards and alarms there, and tap into the full power of Amazon CloudWatch for observability:\n\n\u003cimg src=\"./assets/images/sample-metric.png\" alt=\"Sample Amazon Cloud Metric\" width=\"1200\" /\u003e\n\n## Reference Architecture\n\nThe following is a reference architecture for a setup that uses the Generative AI Toolkit to implement an agent, collect traces, and run automated evaluation. The resulting metrics are fed back to the agent's developers via dashboards and alerts. Metrics are calculated and captured continuously, as real users interact with the agent, thereby giving the agent's developers insight into how the agent is performing at all times, allowing for continuous improvement:\n\n\u003cimg src=\"./assets/images/architecture.drawio.png\" alt=\"Architecture\" width=\"1200\" /\u003e\n\n\u003e Also see our **sample notebook [deploying_on_aws.ipynb](/examples/deploying_on_aws.ipynb)**.\n\n## Key Terms\n\nTo fully utilize the Generative AI Toolkit, it’s essential to understand the following key terms:\n\n- **Traces**: Traces are records of the internal operations of your LLM-based application, e.g. LLM invocations and tool invocations. Traces capture the entire request-response cycle, including input prompts, model outputs, tool calls, and metadata such as latency, token usage, and execution details. Traces form the foundation for evaluating an LLM-based application's behavior and performance.\n\n- **Metrics**: Metrics are measurements derived from traces that evaluate various aspects of an LLM-based application's performance. Examples include latency, token usage, similarity with expected responses, sentiment, and cost. Metrics can be customized to measure specific behaviors or to enforce validation rules.\n\n- **Cases**: Cases are repeatable test inputs that simulate conversations with the agent, e.g. for the purpose of agent evaluation. They consist of a sequence of user inputs and expected agent behaviors or outcomes. Cases are used to validate the agent's responses against defined expectations, ensuring consistent performance across scenarios.\n\n- **Agents**: An agent is an implementation of an LLM-based application that processes user inputs and generates responses. The toolkit provides a simple and extensible agent implementation with built-in support for tracing and tool integration.\n\n- **Tools**: Tools are external functions or APIs that agents can invoke to provide additional capabilities (e.g., fetching weather data or querying a database). Tools are registered with agents and seamlessly integrated into the conversation flow.\n\n- **Conversation History**: This refers to the sequence of messages exchanged between the user and the agent. It can be stored in memory or persisted to external storage, such as DynamoDB, to maintain context across sessions.\n\n- **CloudWatch Custom Metrics**: These are metrics logged to Amazon CloudWatch in Embedded Metric Format (EMF), enabling the creation of dashboards, alarms, and aggregations to monitor agent performance in production environments.\n\n- **Web UI**: A local web-based interface that allows developers to inspect traces, debug conversations, and view evaluation results interactively. This is particularly useful for identifying and resolving issues in the agent's responses.\n\n## Table of Contents\n\n2.1 [Installation](#21-installation) \n2.2 [Agent Implementation](#22-agent-implementation) \n 2.2.1 [Chat with agent](#221-chat-with-agent) \n 2.2.2 [Conversation history](#222-conversation-history) \n 2.2.3 [Reasoning and other Bedrock Converse Arguments](#223-bedrock-converse-arguments) \n 2.2.4 [Tools](#224-tools) \n 2.2.5 [Multi-agent support](#225-multi-agent-support) \n 2.2.6 [Tracing](#226-tracing) \n2.3 [Evaluation Metrics](#23-evaluation-metrics) \n2.4 [Repeatable Cases](#24-repeatable-cases) \n2.5 [Cases with Dynamic Expectations](#25-cases-with-dynamic-expectations) \n2.6 [Generating Traces: Running Cases in Bulk](#26-generating-traces-running-cases-in-bulk) \n2.7 [CloudWatch Custom Metrics](#27-cloudwatch-custom-metrics) \n2.8 [Deploying and Invoking the BedrockConverseAgent](#28-deploying-and-invoking-the-bedrockconverseagent) \n2.9 [Web UI for Conversation Debugging](#29-web-ui-for-conversation-debugging) \n2.10 [Mocking and Testing](#210-mocking-and-testing) \n2.11 [Model Context Protocol (MCP) Client](#211-model-context-protocol-mcp-client)\n\n### 2.1 Installation\n\nInstall `generative_ai_toolkit` with support for all features, amongst which interactive evaluation of metrics:\n\n```bash\npip install \"generative-ai-toolkit[all]\" # Note the [all] modifier\n```\n\nIf you don't use the `[all]` installation modifier, only the minimal set of dependencies will be included that you'll need for creating an agent.\n\nOther available modifiers are:\n\n- `[run-agent]`: includes dependencies such as `gunicorn` that allow you to use `generative_ai_toolkit.run.agent.Runner` to expose your agent over HTTP.\n- `[evaluate]`: includes dependencies that allow you to run evaluations against traces.\n\n### 2.2 Agent implementation\n\nThe heart of the Generative AI Toolkit are the traces it collects, that are the basis for evaluations (explained below). The toolkit includes a simple agent implementation that is backed by the [Amazon Bedrock Converse API](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference.html) and that is instrumented to collect traces in the right format.\n\nA benefit of using this agent implementation, is that you can run the agent locally––it doesn't require any AWS deployment at all and only needs Amazon Bedrock model access. You can quickly iterate and try different agent settings, such as the backing LLM model id, system prompt, temperature, tools, etc. You can create repeatable test cases and run extensive and rigorous evaluations locally.\n\n\u003e We'll first explain how our agent implementation works. Feel free to directly skip to the explanation of [Tracing](#23-tracing) or [Metrics](#24-metrics) instead.\n\nThe Generative AI Toolkit Agent implementation is simple and lightweight, and makes for a no-nonsense developer experience. You can easily instantiate and converse with agents while working in the Python interpreter (REPL) or in a notebook:\n\n```python\nfrom generative_ai_toolkit.agent import BedrockConverseAgent\n\nagent = BedrockConverseAgent(\n model_id=\"anthropic.claude-3-sonnet-20240229-v1:0\",\n)\n```\n\nObviously right now this agent doesn't have any tools yet (we'll add some shortly), but you can already chat with it.\n\n#### 2.2.1 Chat with agent\n\nUse `converse()` to chat with the agent. You pass the user's input to this function, and it will return the agent's response as string:\n\n```python\nresponse = agent.converse(\"What's the capital of France?\")\nprint(response) # \"The capital of France is Paris.\"\n```\n\n##### Response streaming\n\nYou can also use `converse_stream()` to chat with the agent. You pass the user's input to this function, and it will return an iterator that will progressively return the response fragments. You should concatenate these fragments to collect the full response.\n\nThe benefit over using `converse()` is that you can show the user the agent's response tokens as they're being generated, instead of only showing the full response at the very end:\n\n```python\nfor fragment in agent.converse_stream(\"What's the capital of France?\"):\n print(fragment)\n```\n\nThat example might now print several lines to the console, for each set of tokens received, e.g.:\n\n```\nThe\n capital\n of France is\n Paris.\n```\n\n#### 2.2.2 Conversation history\n\nThe agent maintains the conversation history, so e.g. after the question just asked, this would now work:\n\n```python\nresponse = agent.converse(\"What are some touristic highlights there?\") # This goes back to what was said earlier in the conversation\nprint(response) # \"Here are some of the major tourist highlights and attractions in Paris, France:\\n\\n- Eiffel Tower - One of the most famous monuments ...\"\n```\n\nBy default conversation history is stored in memory only. If you want to use conversation history across different process instantiations, you need conversation history that is persisted to durable storage.\n\n##### Persisting conversation history\n\nYou can use the `DynamoDbConversationHistory` class to persist conversations to DynamoDB. Conversation history is maintained per conversation ID. The agent will create a new conversation ID automatically:\n\n```python\nfrom generative_ai_toolkit.agent import BedrockConverseAgent\nfrom generative_ai_toolkit.conversation_history import DynamoDbConversationHistory\n\nagent = BedrockConverseAgent(\n model_id=\"anthropic.claude-3-sonnet-20240229-v1:0\",\n conversation_history=DynamoDbConversationHistory(table_name=\"conversations\") # This table needs to exist, with string keys \"pk\" and \"sk\"\n)\n\nprint(agent.conversation_id) # e.g.: \"01J5D9ZNK5XKZX472HC81ZYR5P\"\n\nagent.converse(\"What's the capital of France?\") # This message, and the agent's response, will now be stored in DynamoDB under conversation ID \"01J5D9ZNK5XKZX472HC81ZYR5P\"\n```\n\nThen later, in another process, if you want to continue this conversation, set the conversation ID first:\n\n```python\nfrom generative_ai_toolkit.agent import BedrockConverseAgent\nfrom generative_ai_toolkit.conversation_history import DynamoDbConversationHistory\n\nagent = BedrockConverseAgent(\n model_id=\"anthropic.claude-3-sonnet-20240229-v1:0\",\n conversation_history=DynamoDbConversationHistory(table_name=\"conversations\")\n)\n\nagent.set_conversation_id(\"01J5D9ZNK5XKZX472HC81ZYR5P\")\n\nresponse = agent.converse(\"What are some touristic highlights there?\")\nprint(response) # \"Here are some of the major tourist highlights and attractions in Paris, France:\\n\\n- Eiffel Tower - One of the most famous monuments ...\"\n```\n\n##### Viewing the conversation history\n\nYou can manually view the conversation history like so:\n\n```python\nprint(agent.messages)\n# [{'role': 'user', 'content': [{'text': \"What's the capital of France?\"}]}, {'role': 'assistant', 'content': [{'text': 'The capital of France is Paris.'}]}, {'role': 'user', 'content': [{'text': 'What are some touristic ...\n```\n\nConversation history is included automatically in the prompt to the LLM. That is, you only have to provide new user input when you call `converse()` (or `converse_stream()`), but under the hood the agent will include all past messages as well.\n\nThis is generally how conversations with LLMs work––the LLM has no memory of the current conversation, you need to provide all past messages, including those from the LLM (the \"assistant\"), as part of your prompt to the LLM.\n\n##### Starting a fresh conversation\n\nCalling `agent.reset()` starts a new conversation, with empty conversation history:\n\n```python\nprint(agent.conversation_id) # e.g.: \"01J5D9ZNK5XKZX472HC81ZYR5P\"\nagent.converse(\"Hi!\")\nprint(len(agent.messages)) # 2 (user input + agent response)\nagent.reset()\nprint(len(agent.messages)) # 0\nprint(agent.conversation_id) # e.g.: \"01J5DQRD864TR3BF314CZK8X5B\" (changed)\n```\n\n##### Multi-modal messages\n\nTo send multi-modal messages (image, video, documents) to the agent, use `add_message()` on the agent's conversation history:\n\n```python\nimage = open(\"/path/to/image\", \"rb\").read()\n\nagent.conversation_history.add_message(\n {\n \"role\": \"user\",\n \"content\": [\n {\"image\": {\"format\": \"png\", \"source\": {\"bytes\": image}}}\n ],\n }\n)\n\n# Then, when you chat with the agent, it will include the message you added to the LLM invocation:\nagent.converse(\"Describe the image please\")\n```\n\n#### 2.2.3 Bedrock Converse Arguments\n\nUpon instantiating the `BedrockConverseAgent` you can pass any arguments that the Bedrock Converse API accepts, and these will be used for all invocations of the Converse API by the agent. You could for example specify usage of [Amazon Bedrock Guardrails](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html):\n\n```python\nfrom generative_ai_toolkit.agent import BedrockConverseAgent\n\nagent = BedrockConverseAgent(\n model_id=\"anthropic.claude-3-sonnet-20240229-v1:0\",\n system_prompt=\"your system prompt\",\n max_tokens=500,\n temperature=0.0,\n top_p=0.8,\n stop_sequences=[\"stop sequence\"],\n guardrail_identifier=\"guardrail-id\",\n guardrail_version=\"guardrail-version\",\n guardrail_trace=\"enabled_full\",\n guardrail_stream_processing_mode=\"async\",\n additional_model_request_fields={\"foo\": \"bar\"},\n prompt_variables={\"foo\": {\"text\": \"bar\"}},\n additional_model_response_field_paths=[\"/path\"],\n request_metadata={\"foo\": \"bar\"},\n performance_config={\"latency\": \"optimized\"},\n)\n```\n\n##### Reasoning\n\nIf you want to use reasoning with a model that supports it (e.g. `anthropic.claude-3-7-sonnet-20250219-v1:0`), specify `additional_model_request_fields`:\n\n```python\nfrom generative_ai_toolkit.agent import BedrockConverseAgent\n\nagent = BedrockConverseAgent(\n model_id=\"anthropic.claude-3-7-sonnet-20250219-v1:0\",\n additional_model_request_fields={\n \"reasoning_config\": {\"type\": \"enabled\", \"budget_tokens\": 1024}\n },\n)\n```\n\nThen, when calling `converse` or `converse_stream`, reasoning texts will be included within `\u003cthinking\u003e` tags in the output:\n\n```python\nresponse = agent.converse(\"How should I make Spaghetti Carbonara?\")\nprint(response)\n```\n\nWould print e.g.:\n\n```\n\u003cthinking\u003e\nThe user is asking for a recipe for Spaghetti Carbonara. I have a tool available called `get_recipe` that can provide recipes.\n\nThe required parameter for this function is:\n- dish: The name of the dish to get a recipe for\n\nIn this case, the dish is \"Spaghetti Carbonara\". This is clearly stated in the user's request, so I can call the function with this parameter.\n\u003c/thinking\u003e\n\nI can help you with a recipe for Spaghetti Carbonara! Let me get that for you.\nHere is the recipe ...\n```\n\nIf you do not want to include the reasoning texts in the output, you can turn that off like so:\n\n```python\nfrom generative_ai_toolkit.agent import BedrockConverseAgent\n\nagent = BedrockConverseAgent(\n model_id=\"anthropic.claude-3-7-sonnet-20250219-v1:0\",\n additional_model_request_fields={\n \"reasoning_config\": {\"type\": \"enabled\", \"budget_tokens\": 1024}\n },\n include_reasoning_text_within_thinking_tags=False, # set this to False\n)\n```\n\n#### 2.2.4 Tools\n\nIf you want to give the agent access to tools, you can define them as Python functions, and register them with the agent. Your Python function must have type annotations for input and output, and a docstring like so:\n\n```python\ndef weather_report(city_name: str) -\u003e str:\n \"\"\"\n Gets the current weather report for a given city\n\n Parameters\n ------\n city_name: string\n The name of the city\n \"\"\"\n return \"Sunny\" # return a string, number, dict or list --\u003e something that can be turned into JSON\n\nagent.register_tool(weather_report)\n\nresponse = agent.converse(\"What's the weather like right now in Amsterdam?\")\nprint(response) # Okay, let me get the current weather report for Amsterdam using the available tool: The weather report for Amsterdam shows that it is currently sunny there.\n```\n\nAs you can see, tools that you've registered will be invoked automatically by the agent. The output from `converse` is always just a string with the agent's response to the user.\n\n##### Multi-modal responses\n\nTools can return multi-modal responses (image, video, documents) as well. If you want that, your tool response should match the format expected by the Amazon Bedrock Converse API:\n\n```python\nfrom mypy_boto3_bedrock_runtime.type_defs import ToolResultContentBlockUnionTypeDef # Optional, to help you with coding\n\ndef get_image() -\u003e list[ToolResultContentBlockUnionTypeDef]:\n \"\"\"\n Read image from disk\n \"\"\"\n\n image = open(\"/path/to/image\", \"rb\").read()\n\n return [{\"image\": {\"format\": \"png\", \"source\": {\"bytes\": image}}}]\n\nagent.register_tool(get_image)\n```\n\nSee more examples in our test suite [here](/tests/integration/test_tool_multi_modal.py).\n\n##### Other tools\n\nIf you don't want to register a Python function as tool, but have a tool with tool spec ready, you can also use it directly, as long as your tool satisfies the `Tool` protocol, i.e. has this shape:\n\n```python\nfrom typing import Any\nfrom mypy_boto3_bedrock_runtime.type_defs import ToolSpecificationTypeDef\n\nclass MyTool:\n\n @property\n def tool_spec(self) -\u003e ToolSpecificationTypeDef:\n return {\"name\":\"my-tool\",\"description\":\"This tool helps with ...\", \"inputSchema\": {...}}\n\n def invoke(self, *args, **kwargs) -\u003e Any:\n return \"Tool response\"\n\nagent.register_tool(MyTool())\n```\n\nIt's also possible to provide the tool spec explicitly alongside your plain Python function:\n\n```python\nagent.register_tool(\n lambda preferred_weather: f\"Not {preferred_weather}\",\n tool_spec={\n \"name\": \"get_weather\",\n \"description\": \"Gets the current weather\",\n \"inputSchema\": {\n \"json\": {\n \"type\": \"object\",\n \"properties\": {\n \"preferred_weather\": {\n \"type\": \"string\",\n \"description\": \"The preferred weather\",\n },\n },\n \"required\": [\n \"preferred_weather\",\n ],\n }\n },\n },\n)\n```\n\n##### Tools override\n\nIt's possible to set and override the tool selection when calling converse:\n\n```python\ndef bad_weather_report(city_name: str) -\u003e str:\n \"\"\"\n Gets the current weather report for a given city\n\n Parameters\n ------\n city_name: string\n The name of the city\n \"\"\"\n return \"Rainy\"\n\nresponse = agent.converse(\"What's the weather like right now in Amsterdam?\", tools=[bad_weather_report])\nprint(response) # Okay, let me check the current weather report for Amsterdam using the available tool:\\nAccording to the tool, the current weather report for Amsterdam is rainy.\n```\n\nNote that this does not force the agent to use the provided tools, it merely makes them available for the agent to use.\n\n##### Tool Development with Pydantic\n\nYou can use Pydantic models to define your tool's interface. This approach provides several key benefits:\n\n1. **Clear Interface Documentation**: Input/output schemas are automatically generated from your models. The LLM \"reads\" both the model's docstring and the `description` attributes of Pydantic Field objects to understand how to use the tool correctly. This natural language documentation helps the LLM make informed decisions about parameter values.\n2. **Error Handling with Self-Correction**: Built-in error handling and validation messages are fed back to the LLM, allowing it to understand what went wrong and self-correct its tool usage. For example, if the LLM provides an invalid value for a parameter, Pydantic's detailed error message helps the LLM understand why it was invalid and how to fix it in subsequent attempts.\n3. **Strong Type Validation**: Pydantic enforces strict type checking and validation at runtime\n\nYou can find a complete example in `examples/pydantic_tools/` that demonstrates this approach. The example implements a weather alerts tool with proper input validation, error handling, and response structuring:\n\n```python\nfrom pydantic import BaseModel, Field\n\nclass WeatherAlertRequest(BaseModel):\n \"\"\"\n Request parameters for the weather alerts tool.\n \"\"\"\n area: Optional[str] = Field(\n default=None,\n description=\"State code (e.g., 'CA', 'TX') or zone/county code to filter alerts by area.\"\n )\n severity: Optional[str] = Field(\n default=None,\n description=\"Filter by severity level: 'Extreme', 'Severe', 'Moderate', 'Minor', or 'Unknown'.\",\n pattern=\"^(Extreme|Severe|Moderate|Minor|Unknown)$\"\n )\n\nclass WeatherAlertsTool:\n @property\n def tool_spec(self) -\u003e Dict[str, Any]:\n \"\"\"Tool specification is automatically generated from the Pydantic model.\"\"\"\n schema = WeatherAlertRequest.model_json_schema()\n return {\n \"name\": \"get_weather_alerts\",\n \"description\": WeatherAlertRequest.__doc__,\n \"inputSchema\": {\"json\": schema}\n }\n\n def invoke(self, **kwargs) -\u003e Dict[str, Any]:\n \"\"\"\n Invoke the weather alerts tool with validated parameters.\n \"\"\"\n try:\n request = WeatherAlertRequest(**kwargs) # Validation happens here\n return self._get_weather_alerts(request)\n except ValidationError as e:\n return {\"error\": str(e)}\n```\n\n##### Tool Registry\n\nYou can organize and discover tools using the `ToolRegistry` and `@tool` decorator. Using the `@tool` decorator can be easier than importing and invoking `agent.register_tool()` for each tool individually.\n\nFor example, let's say you have this in `my_tools/weather.py`:\n\n```python\nfrom generative_ai_toolkit.agent import registry\n\n# Use the decorator to register a function with the default tool registry:\n@registry.tool\ndef get_weather(city: str) -\u003e; str:\n \"\"\"Gets the current weather for a city\"\"\"\n return f\"Sunny in {city}\"\n\n# More tools here, all decorated with @registry.tool\n# ...\n```\n\nYou can then import all modules under `my_tools` and add the tools therein to your agent like so:\n\n```python\nfrom generative_ai_toolkit.agent import BedrockConverseAgent\nfrom generative_ai_toolkit.agent.registry import ToolRegistry, DEFAULT_TOOL_REGISTRY\n\n# You have to import the Python modules with your tools.\n# If they are separate .py files in a local folder,\n# import the folder and all Python modules in it:\nimport my_tools\nToolRegistry.recursive_import(my_tools)\n\n# This would have worked too, without needing recursive import,\n# but would be inconvenient if there's many such modules:\nimport my_tools.weather\n\n# Then, use the populated registry upon creating your agent:\nagent = BedrockConverseAgent(\n model_id=\"anthropic.claude-3-sonnet-20240229-v1:0\",\n tools=DEFAULT_TOOL_REGISTRY,\n)\n```\n\nBy default the `@tool` decorator adds tools to the `DEFAULT_TOOL_REGISTRY` but you can also add them to a custom registry. This can be convenient in a multi-agent scenario:\n\n```python\nfrom generative_ai_toolkit.agent.registry import ToolRegistry, tool\n\n# Create separate registries for different agents:\nweather_registry = ToolRegistry()\nfinance_registry = ToolRegistry()\n\n# Register tools with specific registries:\n@tool(tool_registry=weather_registry)\ndef get_weather_forecast(city: str) -\u003e str:\n \"\"\"Gets the weather forecast for a city\"\"\"\n return f\"Sunny forecast for {city}\"\n\n@tool(tool_registry=finance_registry)\ndef get_stock_price(ticker: str) -\u003e float:\n \"\"\"Gets the current stock price\"\"\"\n return 100.0\n\n# Common tool:\n@tool(tool_registry=[weather_registry, finance_registry])\ndef common_tool(param: str) -\u003e str:\n \"\"\"A common tool that should be available to both agents\"\"\"\n return \"common\"\n\n\n# Create specialized agents with their own tool sets:\nweather_agent = BedrockConverseAgent(\n model_id=\"anthropic.claude-3-haiku-20240307-v1:0\",\n tools=weather_registry,\n)\n\nfinance_agent = BedrockConverseAgent(\n model_id=\"anthropic.claude-3-sonnet-20240229-v1:0\",\n tools=finance_registry,\n)\n```\n\n##### Agent Context in Tools\n\nTools can access contextual information about the current agent execution through the `AgentContext` class, that has the following shape:\n\n```python\nclass AgentContext:\n conversation_id: str\n \"\"\"The conversation ID of the agent\"\"\"\n\n tracer: Tracer\n \"\"\"The tracer that is used by the agent; tools can use it for adding their own traces\"\"\"\n\n auth_context: AuthContext\n \"\"\"The auth context; tools can use it for enforcing authentication and authorization\"\"\"\n\n stop_event: Event\n \"\"\"\n Stop event (threading) that may be set by the user to signal abortion; tools that run for a longer span of time\n should consult the stop event regularly (`stop_event.is_set()`) and abort early if it is set\n \"\"\"\n\n @classmethod\n def current(cls) -\u003e \"AgentContext\":\n \"\"\"\n Access the current agent context from within a tool invocation\n \"\"\"\n ...\n```\n\nExample usage:\n\n```python\nfrom generative_ai_toolkit.context import AgentContext\n\ndef context_aware_tool(some_parameter: str) -\u003e str:\n \"\"\"\n A tool that demonstrates access to agent context\n\n Parameters\n ----------\n some_parameter : str\n Some parameter\n \"\"\"\n\n # Access the current agent context:\n context = AgentContext.current()\n\n # Access conversation and authentication information:\n conversation_id = context.conversation_id\n principal_id = context.auth_context[\"principal_id\"]\n other_auth_data = context.auth_context[\"extra\"][\"other_auth_data\"]\n\n # Access the tracer to be able to use it from within the tool\n # Add attributes to the current span:\n current_trace = context.tracer.current_trace\n current_trace.add_attribute(\"foo\", \"bar\")\n\n # Start a new span:\n with context.tracer.trace(\"new-span\") as trace:\n ...\n\n # Consult the stop event regularly in long running tasks:\n while True:\n if context.stop_event.is_set():\n raise RuntimeException(\"Early abort\")\n ...\n\n return \"response\"\n\nagent.register_tool(context_aware_tool)\n\n# Set context on your agent:\nagent.set_conversation_id(\"01J5D9ZNK5XKZX472HC81ZYR5Z\")\nagent.set_auth_context(principal_id=\"john\", extra={\"other_auth_data\":\"foo\"})\n\n# Now, when the agent invokes the tool during the conversation, the tool can access the context:\nagent.converse(\"Hello!\")\n```\n\nIf you want to use a `stop_event`, create one and pass it to `converse` or `converse_stream`:\n\n```python\nstop_event = threading.Event()\nfor trace in agent.converse_stream(\"Hello again!\", stop_event=stop_event):\n # The stop event that you provided is set onto the agent context\n ...\n\n# At some point in your code, stop the agent and all tool invocations:\nstop_event.set()\n```\n\n##### Testing Tools that Use AgentContext\n\nWhen testing tools that depend on `AgentContext.current()`, you can use the `set_test_context()` helper method to set up test fixtures:\n\n```python\nimport pytest\n\nfrom generative_ai_toolkit.context import AgentContext\n\n@pytest.fixture\ndef agent_context():\n return AgentContext.set_test_context()\n\n# Or with custom values:\n@pytest.fixture\ndef custom_agent_context():\n return AgentContext.set_test_context(\n conversation_id=\"test-conversation\",\n AuthContext(principal_id=\"test\", extras={\"role\": \"admin\"})\n )\n\n# Example tool that uses context\ndef example_tool(message: str) -\u003e str:\n \"\"\"Example tool that accesses agent context\"\"\"\n context = AgentContext.current()\n return f\"User {context.auth_context['principal_id']} says: {message}\"\n\n# Test using the fixture\ndef test_tool_with_context(agent_context):\n result = example_tool(\"Hello\")\n assert \"test\" in result\n assert \"Hello\" in result\n```\n\n#### 2.2.5 Multi-Agent Support\n\nAgents can themselves be used as tool too. This allows you to build hierarchical multi-agent systems, where a supervisor agent can use specialized subordinate agents to delegate tasks to.\n\nTo use an agent as a tool, the agent must have a `name` and `description`:\n\n```python\n# Create a specialized weather agent:\nweather_agent = BedrockConverseAgent(\n model_id=\"anthropic.claude-3-haiku-20240307-v1:0\",\n system_prompt=\"You provide the weather forecast for the specified city.\",\n name=\"transfer_to_weather_agent\", # will be used as the tool name when registered\n description=\"Get the weather forecast for a city.\", # will be used as the tool description\n)\n\n# Add tools to the specialized agent:\ndef get_weather(city: str):\n \"\"\"Gets the weather forecast for the provided city\"\"\"\n return \"Sunny\"\n\nweather_agent.register_tool(get_weather)\n\n# Create a supervisor agent that uses the specialized agent:\nsupervisor = BedrockConverseAgent(\n model_id=\"anthropic.claude-3-sonnet-20240229-v1:0\",\n system_prompt=\"You provide users with information about cities they want to visit.\",\n)\n\n# Register the specialized agent as a tool with the supervisor:\nsupervisor.register_tool(weather_agent)\n\n# The supervisor will delegate to the specialized agent:\nresponse = supervisor.converse(\"What's the weather like in Amsterdam?\")\n```\n\nNotes:\n\n- More layers of nesting can be added if desired; a subordinate agent can itself be supervisor to its own set of subordinate agents, etc.\n- The above example is obviously contrived; for a more comprehensive example with multiple specialized agents working together, see [multi_agent.ipynb](/examples/multi_agent.ipynb).\n\n##### Input schema\n\nBy default, when an agent is used as tool (i.e. as subordinate agent by a supervisor agent), its input schema is:\n\n```json\n{\n \"type\": \"object\",\n \"properties\": {\n \"user_input\": {\n \"type\": \"string\",\n \"description\": \"The input to the agent\"\n }\n },\n \"required\": [\"user_input\"]\n}\n```\n\nNote: the above schema matches the `converse()` method of the `BedrockConverseAgent`, as that will be used under the hood.\n\nIf you want to make sure the agent is called with particular inputs, you can provide an input schema explicitly:\n\n```python\nweather_agent = BedrockConverseAgent(\n model_id=\"anthropic.claude-3-haiku-20240307-v1:0\",\n system_prompt=\"You provide the weather forecast for the specified city.\",\n name=\"transfer_to_weather_agent\", # will be used as the tool name when registered\n description=\"Get the weather forecast for a city.\", # will be used as the tool description\n input_schema={\n \"type\": \"object\",\n \"properties\": {\n \"user_input\": {\n \"type\": \"string\",\n \"description\": \"The city to get the weather for\"\n }\n },\n \"required\": [\"city\"]\n }\n)\n```\n\nThen, when the supervisor invokes the subordinate agent, the supervisor will call the subordinate agent's `converse()` method with `user_input` that includes a (stringified) JSON object, according to the input schema:\n\n```\nYour input is:\n\n{\"city\": \"Amsterdam\"}\n```\n\nSo, the `user_input` to the agent will always be a Python `str`, but using an `input_schema` allows you to 'nudge' the LLM (of the supervisor agent) to include the requested fields explicitly. Alternatively, you could express which fields you require in the subordinate agent's description. Both approaches can work––you'll have to see what works best for your case.\n\n#### 2.2.6 Tracing\n\nYou can make `BedrockConverseAgent` log traces of the LLM and tool calls it performs, by providing a tracer class.\n\nIn the following example, the `InMemoryTracer` is used, which is meant for use during development:\n\n```python\nfrom generative_ai_toolkit.agent import BedrockConverseAgent\nfrom generative_ai_toolkit.conversation_history import DynamoDbConversationHistory\nfrom generative_ai_toolkit.tracer import InMemoryTracer # Import tracer\n\nagent = BedrockConverseAgent(\n model_id=\"anthropic.claude-3-sonnet-20240229-v1:0\",\n conversation_history=DynamoDbConversationHistory(table_name=\"conversations\"),\n tracer=InMemoryTracer, # Add tracer\n)\n```\n\nNow, when you `converse()` with the agent, and the agent calls e.g. LLM and tools, it will log traces. You can inspect these traces like so:\n\n```python\nresponse = agent.converse(\"What's the capital of France?\")\nprint(agent.traces[0])\n```\n\nWill output e.g.:\n\n```python\nTrace(span_name='converse', span_kind='SERVER', trace_id='33185be48ee341d16bf681a552535a4a', span_id='935272e82e76823c', parent_span_id=None, started_at=datetime.datetime(2025, 4, 15, 19, 33, 38, 961, tzinfo=datetime.timezone.utc), ended_at=datetime.datetime(2025, 4, 15, 19, 33, 38, 715109, tzinfo=datetime.timezone.utc), attributes={'ai.trace.type': 'converse', 'ai.conversation.id': '01JRXF2JHXACD860A6P7N0MXER', 'ai.auth.context': None, 'ai.user.input': \"What's the capital of France?\", 'ai.agent.response': 'The capital of France is Paris.'}, span_status='UNSET', resource_attributes={'service.name': 'BedrockConverseAgent'}, scope=generative-ai-toolkit@current)\n```\n\nThat is the root trace of the conversation, that captures user input and agent response. Other traces capture details such as LLM invocations, Tool invocations, usage of conversation history, etc.\n\n##### Available tracers\n\nThe Generative AI Toolkit includes several tracers out-of-the-box, e.g. the `DynamoDBTracer` that saves traces to DynamoDB, and the `OtlpTracer` that sends traces to an OpenTelemetry collector (e.g. to forward them to AWS X-Ray).\n\nFor a full run-down of all out-of-the-box tracers and how to use them, view [examples/tracing101.ipynb](examples/tracing101.ipynb).\n\n##### Open Telemetry\n\nTraces use the [OpenTelemetry \"Span\" model](https://opentelemetry.io/docs/specs/otel/trace/api/#span). That model works at high level by assigning a unique Trace ID to each incoming request (e.g. over HTTP). All actions that are taken while executing that request, are recorded as \"span\" and will have a unique Span ID. Span name, start timestamp, end timestamp, are recorded at span level.\n\nSo, for example, when a user sends a message to an agent, that will start a trace. Then, for every action the agent takes to handle the user's request, a span is recorded. All these spans share the same trace ID, but have a unique span ID. For example, if the agent invokes an LLM or tool, that is recorded as a span. When the agent returns the response to the user, the trace ends, and multiple spans will have been recorded. Often, user and agent will have a conversation that includes multiple turns: the user gives the agent an instruction, the agent asks follow up questions or confirmation, the user gives additional directions, and so forth until the user's intent is fully achieved. Each back-and-forth between user and agent, i.e. each turn in the conversation, is a trace and will have a unique trace ID and (likely) include multiple spans.\n\nIn the OpenTelemetry Span model, information such as \"the model ID used for the LLM invocation\" must be added to a span as attributes. The Generative AI Toolkit uses the following span attributes:\n\n| Attribute Name | Description |\n| ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `ai.trace.type` | Used to identify the type of trace operation being performed. Values: \"conversation-history-list\", \"conversation-history-add\", \"converse\", \"converse-stream\", \"tool-invocation\", \"llm-invocation\" |\n| `ai.conversation.history.implementation` | The string representation of the conversation history implementation being used (e.g. the name of the Python class) |\n| `peer.service` | Indicates the service being interacted with. Values: \"memory:short-term\", \"tool:{tool_name}\", \"llm:{model_id}\" |\n| `ai.conversation.history.messages` | Contains the messages from the conversation history |\n| `ai.conversation.history.message` | Contains a single message being added to the conversation history |\n| `ai.conversation.id` | The unique identifier for the conversation (inheritable attribute) |\n| `ai.auth.context` | The authentication context for the conversation (inheritable attribute) |\n| `ai.tool.name` | Name of the tool being invoked |\n| `ai.tool.use.id` | Unique identifier for the tool usage |\n| `ai.tool.input` | The input parameters provided to the tool |\n| `ai.tool.output` | The response/output from the tool invocation |\n| `ai.tool.error` | Error information if tool invocation fails |\n| `ai.tool.error.traceback` | Full Python traceback information when tool invocation fails |\n| `ai.user.input` | The input provided by the user in the conversation |\n| `ai.llm.request.inference.config` | Configuration settings for the LLM inference |\n| `ai.llm.request.messages` | Messages being sent to the LLM |\n| `ai.llm.request.model.id` | Identifier of the LLM model being used |\n| `ai.llm.request.system` | System prompt or configuration being sent to the LLM |\n| `ai.llm.request.tool.config` | Tool configuration being sent to the LLM |\n| `ai.llm.request.guardrail.config` | Configuration for a guardrail applied during the request. It restricts or modifies the content in messages based on configured criteria. |\n| `ai.llm.request.additional.model.request.fields` | Additional inference parameters specific to the chosen model that extend the standard inference configuration options. |\n| `ai.llm.request.additional.model.response.field.paths` | Specifies additional fields from the model's response to include explicitly, identified by JSON Pointer paths. |\n| `ai.llm.request.prompt.variables` | Variables defined in a prompt resource, mapped to values provided at runtime, used to dynamically customize prompts. |\n| `ai.llm.request.request.metadata` | Custom key-value pairs included for metadata purposes, primarily for filtering and analyzing invocation logs. |\n| `ai.llm.request.performance.config` | Configuration that specifies performance-related settings, such as latency and resource allocation, tailored for specific model invocations. |\n| `ai.llm.response.output` | Output received from the LLM |\n| `ai.llm.response.stop.reason` | Reason why the LLM stopped generating |\n| `ai.llm.response.usage` | Usage metrics from the LLM response |\n| `ai.llm.response.metrics` | Additional metrics from the LLM response |\n| `ai.llm.response.error` | Error information if the LLM request fails |\n| `ai.llm.response.trace` | Trace information |\n| `ai.llm.response.performance.config` | The performance config |\n| `ai.agent.response` | The final concatenated response from the agent |\n| `ai.agent.cycle.nr` | The cycle number during agent conversation processing, indicating which iteration of the conversation loop is being executed |\n| `ai.agent.cycle.response` | The agent's response text for a specific cycle/iteration during conversation processing |\n| `ai.conversation.aborted` | Boolean flag indicating whether the conversation was aborted due to a stop event |\n| `service.name` | Name of the service, set to the class name of the agent |\n\n##### Viewing traces\n\n```python\nfor trace in agent.traces:\n print(trace)\n print()\n```\n\nWould e.g. print:\n\n```python\nTrace(span_name='converse', span_kind='SERVER', trace_id='33185be48ee341d16bf681a552535a4a', span_id='935272e82e76823c', parent_span_id=None, started_at=datetime.datetime(2025, 4, 15, 19, 33, 38, 961, tzinfo=datetime.timezone.utc), ended_at=datetime.datetime(2025, 4, 15, 19, 33, 38, 715109, tzinfo=datetime.timezone.utc), attributes={'ai.trace.type': 'converse', 'ai.conversation.id': '01JRXF2JHXACD860A6P7N0MXER', 'ai.auth.context': None, 'ai.user.input': \"What's the capital of France?\", 'ai.agent.response': 'The capital of France is Paris.'}, span_status='UNSET', resource_attributes={'service.name': 'BedrockConverseAgent'}, scope=generative-ai-toolkit@current)\n\nTrace(span_name='conversation-history-add', span_kind='CLIENT', trace_id='33185be48ee341d16bf681a552535a4a', span_id='ec7c8e79daac9be0', parent_span_id='935272e82e76823c', started_at=datetime.datetime(2025, 4, 15, 19, 33, 38, 1059, tzinfo=datetime.timezone.utc), ended_at=datetime.datetime(2025, 4, 15, 19, 33, 38, 158808, tzinfo=datetime.timezone.utc), attributes={'ai.trace.type': 'conversation-history-add', 'ai.conversation.history.message': {'role': 'user', 'content': [{'text': \"What's the capital of France?\"}]}, 'ai.conversation.history.implementation': 'DynamoDbConversationHistory(table_name=conversations, identifier=None)', 'peer.service': 'memory:short-term', 'ai.conversation.id': '01JRXF2JHXACD860A6P7N0MXER', 'ai.auth.context': None}, span_status='UNSET', resource_attributes={'service.name': 'BedrockConverseAgent'}, scope=generative-ai-toolkit@current)\n\nTrace(span_name='conversation-history-list', span_kind='CLIENT', trace_id='33185be48ee341d16bf681a552535a4a', span_id='f23f49c975823d9d', parent_span_id='935272e82e76823c', started_at=datetime.datetime(2025, 4, 15, 19, 33, 38, 158828, tzinfo=datetime.timezone.utc), ended_at=datetime.datetime(2025, 4, 15, 19, 33, 38, 186879, tzinfo=datetime.timezone.utc), attributes={'ai.trace.type': 'conversation-history-list', 'ai.conversation.history.implementation': 'DynamoDbConversationHistory(table_name=conversations, identifier=None)', 'peer.service': 'memory:short-term', 'ai.conversation.history.messages': [{'role': 'user', 'content': [{'text': \"What's the capital of France?\"}]}], 'ai.conversation.id': '01JRXF2JHXACD860A6P7N0MXER', 'ai.auth.context': None}, span_status='UNSET', resource_attributes={'service.name': 'BedrockConverseAgent'}, scope=generative-ai-toolkit@current)\n\nTrace(span_name='llm-invocation', span_kind='CLIENT', trace_id='33185be48ee341d16bf681a552535a4a', span_id='92ff8f46baa35ec1', parent_span_id='935272e82e76823c', started_at=datetime.datetime(2025, 4, 15, 19, 33, 38, 186905, tzinfo=datetime.timezone.utc), ended_at=datetime.datetime(2025, 4, 15, 19, 33, 38, 686732, tzinfo=datetime.timezone.utc), attributes={'peer.service': 'llm:claude-3-sonnet', 'ai.trace.type': 'llm-invocation', 'ai.llm.request.inference.config': {}, 'ai.llm.request.messages': [{'role': 'user', 'content': [{'text': \"What's the capital of France?\"}]}], 'ai.llm.request.model.id': 'anthropic.claude-3-sonnet-20240229-v1:0', 'ai.llm.request.system': None, 'ai.llm.request.tool.config': None, 'ai.llm.response.output': {'message': {'role': 'assistant', 'content': [{'text': 'The capital of France is Paris.'}]}}, 'ai.llm.response.stop.reason': 'end_turn', 'ai.llm.response.usage': {'inputTokens': 14, 'outputTokens': 10, 'totalTokens': 24}, 'ai.llm.response.metrics': {'latencyMs': 350}, 'ai.conversation.id': '01JRXF2JHXACD860A6P7N0MXER', 'ai.auth.context': None}, span_status='UNSET', resource_attributes={'service.name': 'BedrockConverseAgent'}, scope=generative-ai-toolkit@current)\n\nTrace(span_name='conversation-history-add', span_kind='CLIENT', trace_id='33185be48ee341d16bf681a552535a4a', span_id='f9e6c4ff0254811c', parent_span_id='935272e82e76823c', started_at=datetime.datetime(2025, 4, 15, 19, 33, 38, 686771, tzinfo=datetime.timezone.utc), ended_at=datetime.datetime(2025, 4, 15, 19, 33, 38, 715055, tzinfo=datetime.timezone.utc), attributes={'ai.trace.type': 'conversation-history-add', 'ai.conversation.history.message': {'role': 'assistant', 'content': [{'text': 'The capital of France is Paris.'}]}, 'ai.conversation.history.implementation': 'DynamoDbConversationHistory(table_name=conversations, identifier=None)', 'peer.service': 'memory:short-term', 'ai.conversation.id': '01JRXF2JHXACD860A6P7N0MXER', 'ai.auth.context': None}, span_status='UNSET', resource_attributes={'service.name': 'BedrockConverseAgent'}, scope=generative-ai-toolkit@current)\n\n```\n\nYou can also display traces in a human friendly format:\n\n```python\nfor trace in agent.traces:\n print(trace.as_human_readable())\n```\n\nWhich would print e.g.:\n\n```\n[33185be48ee341d16bf681a552535a4a/root/935272e82e76823c] BedrockConverseAgent SERVER 2025-04-15T19:33:38.000Z - converse (ai.trace.type='converse' ai.conversation.id='01JRXF2JHXACD860A6P7N0MXER' ai.auth.context='null')\n Input: What's the capital of France?\n Response: The capital of France is Paris.\n\n[33185be48ee341d16bf681a552535a4a/935272e82e76823c/ec7c8e79daac9be0] BedrockConverseAgent CLIENT 2025-04-15T19:33:38.001Z - conversation-history-add (ai.trace.type='conversation-history-add' peer.service='memory:short-term' ai.conversation.id='01JRXF2JHXACD860A6P7N0MXER' ai.auth.context='null')\n Message: {'role': 'user', 'content': [{'text': \"What's the capital of France?\"}]}\n\n[33185be48ee341d16bf681a552535a4a/935272e82e76823c/f23f49c975823d9d] BedrockConverseAgent CLIENT 2025-04-15T19:33:38.158Z - conversation-history-list (ai.trace.type='conversation-history-list' peer.service='memory:short-term' ai.conversation.id='01JRXF2JHXACD860A6P7N0MXER' ai.auth.context='null')\n Messages: [{'role': 'user', 'content': [{'text': \"What's the capital of France?\"}]}]\n\n[33185be48ee341d16bf681a552535a4a/935272e82e76823c/92ff8f46baa35ec1] BedrockConverseAgent CLIENT 2025-04-15T19:33:38.186Z - llm-invocation (ai.trace.type='llm-invocation' peer.service='llm:claude-3-sonnet' ai.conversation.id='01JRXF2JHXACD860A6P7N0MXER' ai.auth.context='null')\nLast message: [{'text': \"What's the capital of France?\"}]\n Response: {'message': {'role': 'assistant', 'content': [{'text': 'The capital of France is Paris.'}]}}\n\n[33185be48ee341d16bf681a552535a4a/935272e82e76823c/f9e6c4ff0254811c] BedrockConverseAgent CLIENT 2025-04-15T19:33:38.686Z - conversation-history-add (ai.trace.type='conversation-history-add' peer.service='memory:short-term' ai.conversation.id='01JRXF2JHXACD860A6P7N0MXER' ai.auth.context='null')\n Message: {'role': 'assistant', 'content': [{'text': 'The capital of France is Paris.'}]}\n\n```\n\nOr, as dictionaries:\n\n```python\nfor trace in agent.traces:\n print(trace.as_dict())\n print()\n```\n\nWhich would print e.g.:\n\n```python\n{'span_name': 'converse', 'span_kind': 'SERVER', 'trace_id': '33185be48ee341d16bf681a552535a4a', 'span_id': '935272e82e76823c', 'parent_span_id': None, 'started_at': datetime.datetime(2025, 4, 15, 19, 33, 38, 961, tzinfo=datetime.timezone.utc), 'ended_at': datetime.datetime(2025, 4, 15, 19, 33, 38, 715109, tzinfo=datetime.timezone.utc), 'attributes': {'ai.trace.type': 'converse', 'ai.conversation.id': '01JRXF2JHXACD860A6P7N0MXER', 'ai.auth.context': None, 'ai.user.input': \"What's the capital of France?\", 'ai.agent.response': 'The capital of France is Paris.'}, 'span_status': 'UNSET', 'resource_attributes': {'service.name': 'BedrockConverseAgent'}, 'scope': {'name': 'generative-ai-toolkit', 'version': 'current'}}\n\n{'span_name': 'conversation-history-add', 'span_kind': 'CLIENT', 'trace_id': '33185be48ee341d16bf681a552535a4a', 'span_id': 'ec7c8e79daac9be0', 'parent_span_id': '935272e82e76823c', 'started_at': datetime.datetime(2025, 4, 15, 19, 33, 38, 1059, tzinfo=datetime.timezone.utc), 'ended_at': datetime.datetime(2025, 4, 15, 19, 33, 38, 158808, tzinfo=datetime.timezone.utc), 'attributes': {'ai.trace.type': 'conversation-history-add', 'ai.conversation.history.message': {'role': 'user', 'content': [{'text': \"What's the capital of France?\"}]}, 'ai.conversation.history.implementation': 'DynamoDbConversationHistory(table_name=conversations, identifier=None)', 'peer.service': 'memory:short-term', 'ai.conversation.id': '01JRXF2JHXACD860A6P7N0MXER', 'ai.auth.context': None}, 'span_status': 'UNSET', 'resource_attributes': {'service.name': 'BedrockConverseAgent'}, 'scope': {'name': 'generative-ai-toolkit', 'version': 'current'}}\n\n{'span_name': 'conversation-history-list', 'span_kind': 'CLIENT', 'trace_id': '33185be48ee341d16bf681a552535a4a', 'span_id': 'f23f49c975823d9d', 'parent_span_id': '935272e82e76823c', 'started_at': datetime.datetime(2025, 4, 15, 19, 33, 38, 158828, tzinfo=datetime.timezone.utc), 'ended_at': datetime.datetime(2025, 4, 15, 19, 33, 38, 186879, tzinfo=datetime.timezone.utc), 'attributes': {'ai.trace.type': 'conversation-history-list', 'ai.conversation.history.implementation': 'DynamoDbConversationHistory(table_name=conversations, identifier=None)', 'peer.service': 'memory:short-term', 'ai.conversation.history.messages': [{'role': 'user', 'content': [{'text': \"What's the capital of France?\"}]}], 'ai.conversation.id': '01JRXF2JHXACD860A6P7N0MXER', 'ai.auth.context': None}, 'span_status': 'UNSET', 'resource_attributes': {'service.name': 'BedrockConverseAgent'}, 'scope': {'name': 'generative-ai-toolkit', 'version': 'current'}}\n\n{'span_name': 'llm-invocation', 'span_kind': 'CLIENT', 'trace_id': '33185be48ee341d16bf681a552535a4a', 'span_id': '92ff8f46baa35ec1', 'parent_span_id': '935272e82e76823c', 'started_at': datetime.datetime(2025, 4, 15, 19, 33, 38, 186905, tzinfo=datetime.timezone.utc), 'ended_at': datetime.datetime(2025, 4, 15, 19, 33, 38, 686732, tzinfo=datetime.timezone.utc), 'attributes': {'peer.service': 'llm:claude-3-sonnet', 'ai.trace.type': 'llm-invocation', 'ai.llm.request.inference.config': {}, 'ai.llm.request.messages': [{'role': 'user', 'content': [{'text': \"What's the capital of France?\"}]}], 'ai.llm.request.model.id': 'anthropic.claude-3-sonnet-20240229-v1:0', 'ai.llm.request.system': None, 'ai.llm.request.tool.config': None, 'ai.llm.response.output': {'message': {'role': 'assistant', 'content': [{'text': 'The capital of France is Paris.'}]}}, 'ai.llm.response.stop.reason': 'end_turn', 'ai.llm.response.usage': {'inputTokens': 14, 'outputTokens': 10, 'totalTokens': 24}, 'ai.llm.response.metrics': {'latencyMs': 350}, 'ai.conversation.id': '01JRXF2JHXACD860A6P7N0MXER', 'ai.auth.context': None}, 'span_status': 'UNSET', 'resource_attributes': {'service.name': 'BedrockConverseAgent'}, 'scope': {'name': 'generative-ai-toolkit', 'version': 'current'}}\n\n{'span_name': 'conversation-history-add', 'span_kind': 'CLIENT', 'trace_id': '33185be48ee341d16bf681a552535a4a', 'span_id': 'f9e6c4ff0254811c', 'parent_span_id': '935272e82e76823c', 'started_at': datetime.datetime(2025, 4, 15, 19, 33, 38, 686771, tzinfo=datetime.timezone.utc), 'ended_at': datetime.datetime(2025, 4, 15, 19, 33, 38, 715055, tzinfo=datetime.timezone.utc), 'attributes': {'ai.trace.type': 'conversation-history-add', 'ai.conversation.history.message': {'role': 'assistant', 'content': [{'text': 'The capital of France is Paris.'}]}, 'ai.conversation.history.implementation': 'DynamoDbConversationHistory(table_name=conversations, identifier=None)', 'peer.service': 'memory:short-term', 'ai.conversation.id': '01JRXF2JHXACD860A6P7N0MXER', 'ai.auth.context': None}, 'span_status': 'UNSET', 'resource_attributes': {'service.name': 'BedrockConverseAgent'}, 'scope': {'name': 'generative-ai-toolkit', 'version': 'current'}}\n\n```\n\n##### Streaming traces\n\nWith `converse_stream()` you can iterate over traces in real-time, as they are produced by the agent and its tools. For this, set parameter `stream` to `traces`:\n\n```python\nfor trace in agent.converse_stream(\"What's the capital of France?\", stream=\"traces\"):\n print(trace)\n```\n\nIn `traces` mode, `converse_stream()` yields `Trace` objects as they are generated during the conversation, allowing you to monitor and analyze the agent's behavior as it runs. Each trace contains information about a specific operation (such as LLM invocation, tool usage, etc.) with all relevant attributes like timestamps, inputs, outputs, and more.\n\nThe stream includes both complete traces and trace snapshots. Snapshots represent intermediate traces that are still in progress and can be identified by checking if `trace.ended_at` is `None`.\n\nStreaming traces can be particularly useful for user-facing applications that want to display detailed progress incrementally (like the [chat UI for interactive agent conversations](#chat-ui-for-interactive-agent-conversations)).\n\n##### Multi-Agent Tracing\n\nIn a multi-agent setup, when you access `agent.traces`, this not only returns the traces from the agent itself, but also from all its subagents (recursively). For example, consider this setup:\n\n```\nSupervisorAgent\n├── PlanningAgent\n│ ├── ResearchAgent\n│ ├── DecompositionAgent\n│ └── TimelineAgent\n├── ExecutionAgent\n│ ├── CodingAgent\n│ ├── TestingAgent\n│ └── DeploymentAgent\n└── CommunicationAgent\n ├── UserInteractionAgent\n ├── ReportAgent\n └── FeedbackCollectorAgent\n```\n\nThen:\n\n- If you access `SupervisorAgent.traces`, that would return all traces from all agents in the tree.\n- If you access `PlanningAgent.traces`, that would return the traces from the `PlanningAgent` and its subagents.\n- If you access `ResearchAgent.traces`, that would just return the traces of the `ResearchAgent`.\n\nUnder the hood, this works as follows. When an agent invokes a subagent (as tool), the span id of the tool invocation trace is set onto the subagents trace context as attribute `\"ai.agent.hierarchy.parent.span.id\"`. All traces that are generated by the subagent during that invocation will be \"tagged\" with that attribute value. Then, when the traces of a supervisor agent are accessed (e.g. `SupervisorAgent.traces`), subagent invocations are found too, and all subagent traces that have a `\"ai.agent.hierarchy.parent.span.id\"` matching the tool-invocation span id of the supervisor are included. This is recursive, so if the subagent invoked sub-subagents itself, those would be included too.\n\nSimilarly, when you use `converse_stream(..., stream=\"traces\")`, this yields subagent traces. Conceptually, you could express that as:\n\n```python\nassert list(SupervisorAgent.converse_stream(..., stream=\"traces\")) == SupervisorAgent.traces\n```\n\n##### Web UI\n\nYou can view the traces for a conversation using the Generative AI Toolkit Web UI:\n\n```python\nfrom generative_ai_toolkit.ui import traces_ui\ndemo = traces_ui(agent.traces)\ndemo.launch()\n```\n\nThat opens the Web UI at http://127.0.0.1:7860. E.g. a conversation, that includes an invocation of a weather tool, would look like this:\n\n\u003cimg src=\"./assets/images/ui-traces.png\" alt=\"UI Traces Display Screenshot\" title=\"UI Traces Display\" width=\"1000\"/\u003e\n\nNote that by default only traces for LLM invocations and Tool invocations are shown, as well as user input and agent output. You can choose to view all traces, which would also show e.g. usage of conversational memory, and any other traces the agent developer may have decided to add.\n\nStop the Web UI as follows:\n\n```python\ndemo.close()\n```\n\nNote that you can also use the [chat UI for interactive agent conversations](#chat-ui-for-interactive-agent-conversations), which also shows traces.\n\n##### DynamoDB example\n\nAs example, here's some traces that were stored with the `DynamoDBTracer`:\n\n\u003cimg src=\"./assets/images/dynamodb-traces.png\" alt=\"DynamoDB Traces Display Screenshot\" title=\"DynamoDB Traces Display\" width=\"1200\"/\u003e\n\nIn production deployments, you'll likely want to use the `DynamoDBTracer`, so you can listen to the DynamoDB stream as traces are recorded, and run metric evaluations against them (see next section). This way, you can monitor the performance of your agent in production.\n\n##### AWS X-Ray example\n\nHere's a more elaborate example of a set traces when viewed in AWS X-Ray (you would have used the `OtlpTracer` to send them there):\n\n\u003cimg src=\"./assets/images/x-ray-trace-map.png\" alt=\"AWS X-Ray Trace Map Screenshot\" title=\"AWS X-Ray Trace Map\" width=\"1200\"/\u003e\n\nThe AWS X-Ray view is great because it gives developers an easy-to-digest graphical representation of traces. It's easy to see what the agent did, in which order, how long these actions took, and what the trace attributes are that capture e.g. inputs and outputs for LLM invocations and tool invocations (see the \"Metadata\" pane on the right) :\n\n\u003cimg src=\"./assets/images/x-ray-trace-segments-timeline.png\" alt=\"AWS X-Ray Trace Segments Timeline Screenshot\" title=\"AWS X-Ray Trace Segments Timeline\" width=\"1200\"/\u003e\n\n### 2.3 Evaluation Metrics\n\nMetrics allow you to evaluate your LLM-based application (/agent). The Generative AI Toolkit comes with some metrics out of the box, and makes it easy to develop your own metric as well. Metrics work off of traces, and can measure anything that is represented within the traces.\n\nHere is how you can run metrics against traces.\n\n\u003e Note, this is a contrived example for now; in reality you likely won't run metrics against a single conversation you had with the agent, but against a suite of test cases. Hold tight, that will be explained further below.\n\n```python\nfrom generative_ai_toolkit.evaluate.interactive import GenerativeAIToolkit\nfrom generative_ai_toolkit.metrics.modules.conciseness import AgentResponseConcisenessMetric\nfrom generative_ai_toolkit.metrics.modules.latency import LatencyMetric\n\nresults = GenerativeAIToolkit.eval(\n metrics=[AgentResponseConcisenessMetric(), LatencyMetric()],\n traces=[agent.traces] # pass the traces that were automatically collected by the agent in your conversation with it\n)\n\nresults.summary() # this prints a table with averages to stdout\n```\n\nWould e.g. print:\n\n```\n+-----------------+-----------------+------------------+-------------------------+-------------------------+-----------------------+------------------------+-----------------+-----------------+\n| Avg Conciseness | Avg Latency LLM | Avg Latency TOOL | Avg Latency get_weather | Avg Trace count per run | Avg LLM calls per run | Avg Tool calls per run | Total Nr Passed | Total Nr Failed |\n+-----------------+-----------------+------------------+-------------------------+-------------------------+-----------------------+------------------------+-----------------+-----------------+\n| 8.0 | 1187.0 | 0.0 | 0.0 | 3.0 | 2.0 | 1.0 | 0 | 0 |\n+-----------------+-----------------+------------------+-------------------------+-------------------------+-----------------------+------------------------+-----------------+-----------------+\n```\n\nYou can also access each individual measurement object:\n\n```python\nfor conversation_measurements in results:\n for measurement in conversation_measurements.measurements:\n print(measurement) # measurement concerning all traces in the conversation\n for trace_measurements in conversation_measurements.traces:\n for measurement in trace_measurements.measurements:\n print(measurement) # measurement concerning an individual trace\n```\n\nOr, access the measurements as a (flattened) DataFrame:\n\n```python\ndf = results.details()\ndf.head()\n```\n\nNote that measurements can easily be exported to Amazon CloudWatch as Custom Metrics, which allow you to use Amazon CloudWatch for creating dashboards, aggregations, alarms, etc. See further below.\n\n#### Included metrics\n\nThe following metric are included in the Generative AI Toolkit out-of-the-box.\n\n\u003e Note that some of these metrics can only meaningfully be run during development, because they rely on developer expressed expectations (similar to expectations in a unit test). Developers can express these expectations in cases, explained further below.\n\n| Class name | Description | Usage |\n| ------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- |\n| `metrics.modules.latency.TokensMetric` | Measures number of tokens in LLM invocations (input, output, total) | Development, production |\n| `metrics.modules.similarity.AgentResponseSimilarityMetric` | Measures the cosine similarity between an agent's actual response, and the expected responses that were expressed in the case by the developer. This metric requires cases to have the property `expected_agent_responses_per_turn` specified, which can be provided either during instantiation of the case or with `case.add_turn()` | Development only |\n| `metrics.modules.bleu.BleuMetric` | Similar to the `AgentResponseSimilarityMetric`, but calculates the Bleu score to determine similarity, rather than using cosine similarity | Development only |\n| `metrics.modules.sentiment.SentimentMetric` | Measures the sentiment of the conversation, using Amazon Comprehend. | Development, production |\n| `metrics.modules.latency.LatencyMetric` | Measures the latency of LLM and Tool invocations | Development, production |\n| `metrics.modules.cost.CostMetric` | Measures the cost of LLM invocations | Development, production |\n| `metrics.modules.conversation.ConversationExpectationMetric` | Measures how well the conversation aligns with overall expectations that were expressed by the developer in the case. This metric requires cases to have the property `overall_expectations` which can be provided during instantiation of the case. | Development only |\n| `metrics.modules.conciseness.AgentResponseConcisenessMetric` | Measures how concise the agent's response are, i.e. to aid in building agents that don't ramble. This metric is implemented as an LLM-as-judge: an LLM is used to grade the conciseness of the agent's response on a scale from 1 to 10. | Development, production |\n\n#### Custom metrics\n\nLet's now see how you create a custom metric. Here is a custom metric that would measure how many tools the agent actually used in the conversation with the user:\n\n```python\nfrom generative_ai_toolkit.metrics import BaseMetric, Measurement, Unit\n\n\nclass NumberOfToolsUsedMetric(BaseMetric):\n def evaluate_conversation(self, conversation_traces, **kwargs):\n return Measurement(\n name=\"NumberOfToolsUsed\",\n value=len([trace for trace in conversation_traces if trace.attributes.get(\"ai.trace.type\") == \"tool-invocation\"]),\n unit=Unit.Count,\n )\n```\n\nThe above metric works at conversation level and therefore implements `evaluate_conversation` which gets all the traces from the conversation in one go.\n\nEven more simple custom metrics would work at individual trace level, without needing to know about the other traces in the conversation. In that case, implement `evaluate_trace`:\n\n\u003e Note the `TokensMetric` actually comes out-of-the-box, but we'll reimplement it here for sake of the example\n\n```python\nfrom generative_ai_toolkit.metrics import BaseMetric, Measurement, Unit\n\n\nclass TokenCount(BaseMetric):\n if trace.attributes.get(\"ai.trace.type\") != \"llm-invocation\":\n return\n\n input_tokens = trace.attributes[\"ai.llm.response.usage\"][\"inputTokens\"]\n output_tokens = trace.attributes[\"ai.llm.response.usage\"][\"outputTokens\"]\n\n return [\n Measurement(\n name=\"TotalTokens\",\n value=input_tokens + output_tokens,\n unit=Unit.Count,\n ),\n Measurement(\n name=\"InputTokens\",\n value=input_tokens,\n unit=Unit.Count,\n ),\n Measurement(\n name=\"OutputTokens\",\n value=output_tokens,\n unit=Unit.Count,\n ),\n ]\n```\n\nThe above custom metric returns 3 measurements, but only for LLM traces.\n\nEvaluating your own custom metrics works the same as for the out-of-the-box metrics (and they can be matched freely):\n\n```python\nresults = GenerativeAIToolkit.eval(\n metrics=[NumberOfToolsUsedMetric(), TokenCount()],\n traces=[agent.traces]\n)\nresults.summary()\n```\n\nWould e.g. print:\n\n```\n+------------------------------+----------------------+----------------------+-------------------------+-----------------------+------------------------+-----------------+-----------------+\n| Avg NumberOfToolsUsed | Avg NrOfOInputTokens | Avg NrOfOutputTokens | Avg Trace count per run | Avg LLM calls per run | Avg Tool calls per run | Total Nr Passed | Total Nr Failed |\n+------------------------------+----------------------+----------------------+-------------------------+-----------------------+------------------------+-----------------+-----------------+\n| 1.0 | 371.0 | 42.5 | 3.0 | 2.0 | 1.0 | 0 | 0 |\n+------------------------------+----------------------+----------------------+-------------------------+-----------------------+------------------------+-----------------+-----------------+\n```\n\n#### Template for Custom Metrics\n\nUse [TEMPLATE_metric.py](src/generative_ai_toolkit/metrics/modules/TEMPLATE_metric.py) as a starting point for creating your own custom metrics. This file includes more information on the data model, as well as more examples.\n\n#### Passing or Failing a Custom Metric\n\nBesides measuring an agent's performance in a scalar way, custom metrics can (optionally) return a Pass or Fail indicator. This will be reflected in the measurements summary and such traces would be marked as failed in the Web UI for conversation debugging (see further).\n\nLet's tweak our `TokenCount` metric to make it fail if the LLM returns more than 100 tokens:\n\n```python\nfrom generative_ai_toolkit.metrics import BaseMetric, Measurement, Unit\n\n\nclass TokenCount(BaseMetric):\n def evaluate_trace(self, trace, **kwargs):\n if trace.attributes.get(\"ai.trace.type\") != \"llm-invocation\":\n return\n\n input_tokens = trace.attributes[\"ai.llm.response.usage\"][\"inputTokens\"]\n output_tokens = trace.attributes[\"ai.llm.response.usage\"][\"outputTokens\"]\n\n return [\n Measurement(\n name=\"TotalTokens\",\n value=input_tokens + output_tokens,\n unit=Unit.Count,\n ),\n Measurement(\n name=\"InputTokens\",\n value=input_tokens,\n unit=Unit.Count,\n ),\n Measurement(\n name=\"OutputTokens\",\n value=output_tokens,\n unit=Unit.Count,\n validation_passed=output_tokens \u003c= 100, # added, just an example\n ),\n ]\n```\n\nAnd run evaluation again:\n\n```python\nresults = GenerativeAIToolkit.eval(\n metrics=[TokenCount()],\n traces=[agent.traces]\n)\nresults.summary()\n```\n\nWould now e.g. print (note `Total Nr Passed` and `Total Nr Failed`):\n\n```\n+----------------------+----------------------+-------------------------+-----------------------+------------------------+-----------------+-----------------+\n| Avg NrOfOInputTokens | Avg NrOfOutputTokens | Avg Trace count per run | Avg LLM calls per run | Avg Tool calls per run | Total Nr Passed | Total Nr Failed |\n+----------------------+----------------------+-------------------------+-----------------------+------------------------+-----------------+-----------------+\n| 371.5 | 31.0 | 3.0 | 2.0 | 1.0 | 1 | 1 |\n+----------------------+----------------------+-------------------------+-----------------------+------------------------+-----------------+-----------------+\n```\n\n#### Additional information\n\nYou can attach additional information to the measurements you create. This information will be visible in the Web UI for conversation debugging, as well as in Amazon CloudWatch (if you use the seamless export of the measurements to CloudWatch, see further below):\n\n```python\nfrom generative_ai_toolkit.metrics import BaseMetric, Measurement, Unit\n\n\nclass MyMetric(BaseMetric):\n def evaluate_trace(self, trace, **kwargs):\n return Measurement(\n name=\"MyMeasurementName\",\n value=123.456,\n unit=Unit.Count,\n additional_information={\n \"context\": \"This is some context\",\n \"you\": [\"can store\", \"anything\", \"here\"]\n }\n )\n```\n\n### 2.4 Repeatable Cases\n\nYou can create repeatable cases to run against your LLM application. The process is this:\n\n```mermaid\nflowchart LR\n A[\"Create LLM application (agent)\"]\n B[Creates cases]\n C[\"Generate traces by running the cases against the LLM application (agent)\"]\n D[Evaluate the traces with metrics]\n A --\u003e B --\u003e C --\u003e D\n```\n\nA case has a name (optional) and user inputs. Each user input will be fed to the agent sequentially in the same conversation:\n\n```python\nmy_case = Case(\n name=\"User wants to do something fun\",\n user_inputs=[\n \"I wanna go somewhere fun\",\n \"Within 60 minutes\",\n \"A museum of modern art\",\n ],\n)\n```\n\nA case can be run against an agent like this, returning the traces collected:\n\n```python\ntraces = my_case.run(agent)\n```\n\nThat will play out the conversation, feeding each input to the agent, awaiting its response, and then feeding the nextm until all user inputs have been fed to the agent. For quick tests this works, but if you have many cases you'll want to use `generate_traces()` (see below) to run them parallelized in bulk.\n\n#### Cases with expectations\n\nHere is a case with overall expectations, that will be interpreted by the `ConversationExpectationMetric` (if you include that metric upon calling `GenerativeAIToolkit.eval()` against the collected traces):\n\n```python\nimport textwrap\n\n\nconv_expectation_case = Case(\n name=\"User wants to go MoMA\",\n user_inputs=[\n \"I wanna go somewhere fun\",\n \"Within 60 minutes\",\n \"A museum of modern art\",\n ],\n overall_expectations=textwrap.dedent(\n \"\"\"\n The agent first asks the user (1) what type of activity they want to do and (2) how long they're wiling to drive to get there.\n When the user only answers the time question (2), the agent asks the user again what type of activity they want to do (1).\n Then, when the user finally answers the wat question also (1), the agent makes some relevant recommendations, and asks the user to pick.\n \"\"\"\n ),\n)\n```\n\nHere is a case with expectations per turn, that will be interpreted by the `AgentResponseSimilarityMetric` and `BleuMetric` (if you include any of these metrics upon calling `GenerativeAIToolkit.eval()` against the collected traces):\n\n```python\nsimilarity_case = Case(\n name=\"User wants to go to a museum\",\n)\nsimilarity_case.add_turn(\n \"I want to do something fun\",\n [\n \"To help you I need more information. What type of activity do you want to do and how long are you willing to drive to get there?\",\n \"Okay, to find some fun activities for you, I'll need a bit more information first. What kind of things are you interested in doing? Are you looking for outdoor activities, cultural attractions, dining, or something else? And how much time are you willing to spend driving to get there?\",\n ],\n)\nsimilarity_case.add_turn(\n \"I'm thinking of going to a museum\",\n [\n \"How long are you willing to drive to get there?\"\n \"Got it, you're interested in visiting a museum. That's helpful to know. What's the maximum amount of time you're willing to drive to get to the museum?\"\n ],\n)\n```\n\n#### Cases with dynamic input\n\nInstead of listing out all user inputs beforehand, you can provide a user input producer to a case, which is a python function that dynamically creates user inputs to match the conversation. This can be of use during development, to e.g. do smoke tests to get a sense for how well the agent works.\n\nThe `user_input_producer` should be passed to the `Case` and it must be a Python `Callable` that accepts the parameter `messages`, which contains the conversation history. The `user_input_producer` should return new user input each time it's called, or an empty string to signal the conversation should end.\n\nYou can create your own user input producer implementation, or use the out-of-the-box `UserInputProducer` that uses an LLM under the hood to determine the next user utterance:\n\n```python\nfrom generative_ai_toolkit.agent import BedrockConverseAgent\nfrom generative_ai_toolkit.test import Case, UserInputProducer\n\nagent = BedrockConverseAgent(\n model_id=\"anthropic.claude-3-sonnet-20240229-v1:0\",\n system_prompt=\"You help users with movie suggestions. You are succinct and to-the-point\"\n)\n\ndef get_movie_suggestion(genre: str):\n \"\"\"\n Generates a random movie suggestion, for the provided genre.\n Returns one movie suggestion (title) without any further information.\n Ensure the user provides a genre, do not assume the genre––ask the user if not provided.\n\n\n Parameters\n ----------\n genre : str\n The genre of the movie to be suggested.\n \"\"\"\n return \"The alleyways of Amsterdam (1996)\"\n\nagent.register_tool(get_movie_suggestion)\n\n# This case does not have user inputs, but rather a user_input_producer,\n# in this case the UserInputProducer class, which should be instantiated with the user's intent:\ncase = Case(name=\"User wants a movie suggestion\", user_input_producer=UserInputProducer(user_intent=\"User wants a movie suggestion\"))\n\ntraces = case.run(agent)\n\nfor trace in traces:\n print(trace.as_human_readable())\n```\n\nWould print e.g.:\n\n```\n[120027b89023dd54f59c50499b57b599/root/9e22ad550295191f] BedrockConverseAgent SERVER 2025-04-16T09:01:10.466Z - converse (ai.trace.type='converse' ai.conversation.id='01JRYX9AZGZQNYJTQ4V4T3SCGJ' ai.auth.context='null')\n Input: I'd like to get a movie recommendation. What genres or types of movies do you have suggestions for?\n Response: I can provide movie suggestions for different genres. What genre would you like a recommendation for? Examples of genres are action, comedy, drama, romance, horror, sci-fi, etc.\n\n[120027b89023dd54f59c50499b57b599/9e22ad550295191f/1b2b94552c644558] BedrockConverseAgent CLIENT 2025-04-16T09:01:10.467Z - conversation-history-add (ai.trace.type='conversation-history-add' peer.service='memory:short-term' ai.conversation.id='01JRYX9AZGZQNYJTQ4V4T3SCGJ' ai.auth.context='null')\n Message: {'role': 'user', 'content': [{'text': \"I'd like to get a movie recommendation. What genres or types of movies do you have suggestions for?\"}]}\n\n[120027b89023dd54f59c50499b57b599/9e22ad550295191f/19c00ed498f0abee] BedrockConverseAgent CLIENT 2025-04-16T09:01:10.467Z - conversation-history-list (ai.trace.type='conversation-history-list' peer.service='memory:short-term' ai.conversation.id='01JRYX9AZGZQNYJTQ4V4T3SCGJ' ai.auth.context='null')\n Messages: [{'role': 'user', 'content': [{'text': \"I'd like to get a movie recommendation. What genres or types of movies do you have suggestions for?\"}]}]\n\n[120027b89023dd54f59c50499b57b599/9e22ad550295191f/fece03d8bc85f4d8] BedrockConverseAgent CLIENT 2025-04-16T09:01:10.467Z - llm-invocation (ai.trace.type='llm-invocation' peer.service='llm:claude-3-sonnet' ai.conversation.id='01JRYX9AZGZQNYJTQ4V4T3SCGJ' ai.auth.context='null')\nLast message: [{'text': \"I'd like to get a movie recommendation. What genres or types of movies do you have suggestions for?\"}]\n Response: {'message': {'role': 'assistant', 'content': [{'text': 'I can provide movie suggestions for different genres. What genre would you like a recommendation for? Examples of genres are action, comedy, dra\n ma, romance, horror, sci-fi, etc.'}]}}\n\n[120027b89023dd54f59c50499b57b599/9e22ad550295191f/00bf227f03f6f4ae] BedrockConverseAgent CLIENT 2025-04-16T09:01:11.613Z - conversation-history-add (ai.trace.type='conversation-history-add' peer.service='memory:short-term' ai.conversation.id='01JRYX9AZGZQNYJTQ4V4T3SCGJ' ai.auth.context='null')\n Message: {'role': 'assistant', 'content': [{'text': 'I can provide movie suggestions for different genres. What genre would you like a recommendation for? Examples of genres are action, comedy, drama, romance,\n horror, sci-fi, etc.'}]}\n\n[3fdef11a72df06eb74fba3d65402d0da/root/f6a5671219710173] BedrockConverseAgent SERVER 2025-04-16T09:01:12.835Z - converse (ai.trace.type='converse' ai.conversation.id='01JRYX9AZGZQNYJTQ4V4T3SCGJ' ai.auth.context='null')\n Input: I'm interested in comedies. Do you have any good comedy movie suggestions?\n Response: The comedy movie suggestion is \"The Alleyways of Amsterdam\" from 1996. It sounds like an offbeat, quirky comedy set in the Netherlands. Let me know if you'd like another comedy recommendation or if th\n at piqued your interest!\n\n[3fdef11a72df06eb74fba3d65402d0da/f6a5671219710173/fe6efe3b5c21310d] BedrockConverseAgent CLIENT 2025-04-16T09:01:12.835Z - conversation-history-add (ai.trace.type='conversation-history-add' peer.service='memory:short-term' ai.conversation.id='01JRYX9AZGZQNYJTQ4V4T3SCGJ' ai.auth.context='null')\n Message: {'role': 'user', 'content': [{'text': \"I'm interested in comedies. Do you have any good comedy movie suggestions?\"}]}\n\n[3fdef11a72df06eb74fba3d65402d0da/f6a5671219710173/6a1566cb25737d02] BedrockConverseAgent CLIENT 2025-04-16T09:01:12.835Z - conversation-history-list (ai.trace.type='conversation-history-list' peer.service='memory:short-term' ai.conversation.id='01JRYX9AZGZQNYJTQ4V4T3SCGJ' ai.auth.context='null')\n Messages: [{'role': 'user', 'content': [{'text': \"I'd like to get a movie recommendation. What genres or types of movies do you have suggestions for?\"}]}, {'role': 'assistant', 'content': [{'text': 'I can provi\n de movie suggestions for different genres. What genre would you like a recommendation for? Examples of genres are action, comedy, drama, romance, horror, sci-fi, etc.'}]}, {'role': 'user', 'content':\n [{'text': \"I'm interested in comedies. Do you have any good comedy movie suggestions?\"}]}]\n\n[3fdef11a72df06eb74fba3d65402d0da/f6a5671219710173/4723b2cb16046645] BedrockConverseAgent CLIENT 2025-04-16T09:01:12.835Z - llm-invocation (ai.trace.type='llm-invocation' peer.service='llm:claude-3-sonnet' ai.conversation.id='01JRYX9AZGZQNYJTQ4V4T3SCGJ' ai.auth.context='null')\nLast message: [{'text': \"I'm interested in comedies. Do you have any good comedy movie suggestions?\"}]\n Response: {'message': {'role': 'assistant', 'content': [{'toolUse': {'toolUseId': 'tooluse_Tf776MWLQ_iIyuYGsdvvTw', 'name': 'get_movie_suggestion', 'input': {'genre': 'comedy'}}}]}}\n\n[3fdef11a72df06eb74fba3d65402d0da/f6a5671219710173/b49ca36d024d0150] BedrockConverseAgent CLIENT 2025-04-16T09:01:13.832Z - conversation-history-add (ai.trace.type='conversation-history-add' peer.service='memory:short-term' ai.conversation.id='01JRYX9AZGZQNYJTQ4V4T3SCGJ' ai.auth.context='null')\n Message: {'role': 'assistant', 'content': [{'toolUse': {'toolUseId': 'tooluse_Tf776MWLQ_iIyuYGsdvvTw', 'name': 'get_movie_suggestion', 'input': {'genre': 'comedy'}}}]}\n\n[3fdef11a72df06eb74fba3d65402d0da/f6a5671219710173/bcc78bb3c1c1a110] BedrockConverseAgent CLIENT 2025-04-16T09:01:13.832Z - get_movie_suggestion (ai.trace.type='tool-invocation' peer.service='tool:get_movie_suggestion' ai.conversation.id='01JRYX9AZGZQNYJTQ4V4T3SCGJ' ai.auth.context='null')\n Input: {'genre': 'comedy'}\n Output: The alleyways of Amsterdam (1996)\n\n[3fdef11a72df06eb74fba3d65402d0da/f6a5671219710173/c89ae710cadf9952] BedrockConverseAgent CLIENT 2025-04-16T09:01:13.832Z - conversation-history-add (ai.trace.type='conversation-history-add' peer.service='memory:short-term' ai.conversation.id='01JRYX9AZGZQNYJTQ4V4T3SCGJ' ai.auth.context='null')\n Message: {'role': 'user', 'content': [{'toolResult': {'toolUseId': 'tooluse_Tf776MWLQ_iIyuYGsdvvTw', 'status': 'success', 'content': [{'json': {'toolResponse': 'The alleyways of Amsterdam (1996)'}}]}}]}\n\n[3fdef11a72df06eb74fba3d65402d0da/f6a5671219710173/a83184f18d57e2e0] BedrockConverseAgent CLIENT 2025-04-16T09:01:13.832Z - conversation-history-list (ai.trace.type='conversation-history-list' peer.service='memory:short-term' ai.conversation.id='01JRYX9AZGZQNYJTQ4V4T3SCGJ' ai.auth.context='null')\n Messages: [{'role': 'user', 'content': [{'text': \"I'd like to get a movie recommendation. What genres or types of movies do you have suggestions for?\"}]}, {'role': 'assistant', 'content': [{'text': 'I can provi\n de movie suggestions for different genres. What genre would you like a recommendation for? Examples of genres are action, comedy, drama, romance, horror, sci-fi, etc.'}]}, {'role': 'user', 'content':\n [{'text': \"I'm interested in comedies. Do you have any good comedy movie suggestions?\"}]}, {'role': 'assistant', 'content': [{'toolUse': {'toolUseId': 'tooluse_Tf776MWLQ_iIyuYGsdvvTw', 'name': 'get_mo\n vie_suggestion', 'input': {'genre': 'comedy'}}}]}, {'role': 'user', 'content': [{'toolResult': {'toolUseId': 'tooluse_Tf776MWLQ_iIyuYGsdvvTw', 'status': 'success', 'content': [{'json': {'toolRespon...\n\n[3fdef11a72df06eb74fba3d65402d0da/f6a5671219710173/b6a64ab8eb8f4cfc] BedrockConverseAgent CLIENT 2025-04-16T09:01:13.832Z - llm-invocation (ai.trace.type='llm-invocation' peer.service='llm:claude-3-sonnet' ai.conversation.id='01JRYX9AZGZQNYJTQ4V4T3SCGJ' ai.auth.context='null')\nLast message: [{'toolResult': {'toolUseId': 'tooluse_Tf776MWLQ_iIyuYGsdvvTw', 'status': 'success', 'content': [{'json': {'toolResponse': 'The alleyways of Amsterdam (1996)'}}]}}]\n Response: {'message': {'role': 'assistant', 'content': [{'text': 'The comedy movie suggestion is \"The Alleyways of Amsterdam\" from 1996. It sounds like an offbeat, quirky comedy set in the Netherlands. Let me k\n now if you\\'d like another comedy recommendation or if that piqued your interest!'}]}}\n\n[3fdef11a72df06eb74fba3d65402d0da/f6a5671219710173/1ade2dbc33d3db6a] BedrockConverseAgent CLIENT 2025-04-16T09:01:15.410Z - conversation-history-add (ai.trace.type='conversation-history-add' peer.service='memory:short-term' ai.conversation.id='01JRYX9AZGZQNYJTQ4V4T3SCGJ' ai.auth.context='null')\n Message: {'role': 'assistant', 'content': [{'text': 'The comedy movie suggestion is \"The Alleyways of Amsterdam\" from 1996. It sounds like an offbeat, quirky comedy set in the Netherlands. Let me know if you\\'\n d like another comedy recommendation or if that piqued your interest!'}]}\n```\n\nWhat you can see is that the agent asked the user a question because it needed more information (the genre, see first `SERVER` trace), and the user input producer provided an answer on behalf of the user: comedy (see second `SERVER` trace).\n\nNote that you can still provide `user_inputs` in the case as well: these will be played out first, and once these are exhausted the `user_input_producer` will be invoked for getting subsequent user inputs. This way, you can 'prime' a conversation.\n\n### 2.5 Cases with dynamic expectations\n\nCases can also be validated by passing it one or more validator functions. A validator function must be a Python `Callable` that accepts as input the traces of the conversation. Based on these traces the validator function should return `None` or an empty string, if the test passes. If the test fails it should return one or more messages (`str` or `Sequence[str]`).\n\nThe validator function will be invoked when the traces of the case are ran through `GenerativeAIToolkit.eval()` and this will generate measurements automatically: measurements with name `ValidationPassed` if the test passed (i.e. it returned `None` or `\"\"`) and `ValidationFailed` otherwise. If the validation failed, the message that was returned will be included in the measurement's `additional_info` (or if an exception was thrown, the exception message):\n\n```python\ndef validate_weather_report(traces: Sequence[CaseTrace]):\n root_trace = traces[0]\n last_output = root_trace.attributes[\"ai.agent.response\"]\n if last_output.startswith(\"The weather will be\"):\n # Test passed!\n return\n return f\"Unexpected message: {last_output}\"\n\n\ncase1 = Case(\n name=\"Check weather\",\n user_inputs=[\"What is the weather like right now?\"],\n validate=validate_weather_report,\n)\n\ntraces = case1.run(agent)\n\n# To run the validator functions, run GenerativeAIToolkit.eval()\n# Validator functions will be run always, even if no metrics are provided otherwise:\nresults = GenerativeAIToolkit.eval(metrics=[], traces=[traces])\n\nresults.summary()\n\nfor conversation_measurements in results:\n for measurement in conversation_measurements.measurements:\n print(measurement)\n```\n\nThat would e.g. print one failure (if the case has at least one failed validation, it is counted as a failure) and corresponding measurement:\n\n```\n+----------------------+-------------------------+-----------------------+------------------------+-----------------+-----------------+\n| Avg ValidationFailed | Avg Trace count per run | Avg LLM calls per run | Avg Tool calls per run | Total Nr Passed | Total Nr Failed |\n+----------------------+-------------------------+-----------------------+------------------------+-----------------+-----------------+\n| 1.0 | 3.0 | 2.0 | 1.0 | 0 | 1 |\n+----------------------+-------------------------+-----------------------+------------------------+-----------------+-----------------+\n\nMeasurement(name='ValidationFailed', value=1, unit=\u003cUnit.None_: 'None'\u003e, additional_info={'validation_messages': ['Unexpected message: The current weather is sunny. Let me know if you need any other weather details!']}, dimensions=[], validation_passed=False)\n```\n\n### 2.6 Generating traces: running cases in bulk\n\nWhen you have many cases, instead of calling `case.run(agent)` for each case, it's better to run cases in parallel like so:\n\n```python\nfrom generative_ai_toolkit.evaluate.interactive import GenerativeAIToolkit, Permute\n\n\ntraces = GenerativeAIToolkit.generate_traces(\n cases=cases, # pass in an array of cases here\n nr_runs_per_case=3, # nr of times to run each case, to account for LLM indeterminism\n agent_factory=BedrockConverseAgent, # This can also be your own factory function\n agent_parameters={\n \"system_prompt\": Permute(\n [\n \"You are a helpful assistant\",\n \"You are a lazy assistant who prefers to joke around rather than to help users\",\n ]\n ),\n \"temperature\": 0.0,\n \"tools\": my_tools, # list of python functions that can be used as tools\n \"model_id\": Permute(\n [\n \"anthropic.claude-3-sonnet-20240229-v1:0\",\n \"anthropic.claude-3-haiku-20240307-v1:0\",\n ]\n ),\n },\n)\n```\n\nExplanation:\n\n- `generate_traces()` is in essence nothing but a parallelized (with threads) invocation of `case.run(agent)` for each case provided. To account for LLM indeterminism, each case is run `nr_runs_per_case` times.\n- Because an agent instantiation can only handle one conversation at a time, you must pass an `agent_factory` to `generate_traces()` so that it can create a fresh agent instance for each test conversation that it will run through. The `agent_factory` must be a python callable that can be fed `agent_parameters` and returns an agent instance. This can be a `BedrockConverseAgent` as above, but may be any Python object that exposes a `converse` method and `traces` property.\n- The (optional) `agent_parameters` will be supplied to the `agent_factory` you provided.\n- By using `Permute` for values within the `agent_parameters` you can test different parameter values against each other. In the example above, 2 different system prompts are tried, and 2 different model ID's. This in effect means 4 permutations (2 x 2) will be tried, i.e. the full cartes","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fawslabs%2Fgenerative-ai-toolkit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fawslabs%2Fgenerative-ai-toolkit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fawslabs%2Fgenerative-ai-toolkit/lists"}