{"id":15138946,"url":"https://github.com/uezo/aiavatarkit","last_synced_at":"2026-04-02T14:07:55.965Z","repository":{"id":170078689,"uuid":"646174248","full_name":"uezo/aiavatarkit","owner":"uezo","description":"🥰 Building AI-based conversational avatars lightning fast ⚡️💬","archived":false,"fork":false,"pushed_at":"2026-03-24T18:03:07.000Z","size":46165,"stargazers_count":565,"open_issues_count":5,"forks_count":54,"subscribers_count":5,"default_branch":"main","last_synced_at":"2026-03-25T14:14:58.100Z","etag":null,"topics":["chatgpt","metaverse","voicevox","vr","vrchat","waifu"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/uezo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-05-27T14:22:56.000Z","updated_at":"2026-03-24T21:24:05.000Z","dependencies_parsed_at":"2025-12-19T21:00:34.615Z","dependency_job_id":null,"html_url":"https://github.com/uezo/aiavatarkit","commit_stats":{"total_commits":112,"total_committers":1,"mean_commits":112.0,"dds":0.0,"last_synced_commit":"0751ab8294dc38ce2fa340f7a8ba86746eaf7e00"},"previous_names":["uezo/aiavatarkit"],"tags_count":51,"template":false,"template_full_name":null,"purl":"pkg:github/uezo/aiavatarkit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uezo%2Faiavatarkit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uezo%2Faiavatarkit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uezo%2Faiavatarkit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uezo%2Faiavatarkit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/uezo","download_url":"https://codeload.github.com/uezo/aiavatarkit/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/uezo%2Faiavatarkit/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31307542,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-02T12:59:32.332Z","status":"ssl_error","status_checked_at":"2026-04-02T12:54:48.875Z","response_time":89,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatgpt","metaverse","voicevox","vr","vrchat","waifu"],"created_at":"2024-09-26T08:00:38.738Z","updated_at":"2026-04-02T14:07:55.949Z","avatar_url":"https://github.com/uezo.png","language":"Python","funding_links":[],"categories":["Chatbots"],"sub_categories":[],"readme":"# AIAvatarKit\n\n🥰 Building AI-based conversational avatars lightning fast ⚡️💬\n\n![AIAvatarKit Architecture Overview](documents/images/aiavatarkit_overview.png) \n\n## ✨ Features\n\n- **🌏 Live anywhere**: AIAvatarKit is a general-purpose Speech-to-Speech framework with multimodal input/output support. It can serve as the backend for a wide range of conversational AI systems.\n    - Metaverse Platforms: Compatible with VRChat, cluster, Vket Cloud, and other platforms\n    - Standalone Apps: Enables ultra-low latency real-time interaction via WebSocket or HTTP (SSE), with a unified interface that abstracts differences between LLMs\n    - Channels and Devices: Supports edge devices like Raspberry Pi and telephony services like Twilio\n- **🧩 Modular architecture**: Components such as VAD, STT, LLM, and TTS are modular and easy to integrate via lightweight interfaces. Supported modules include:\n    - VAD: Built-in standard VAD (silence-based end-of-turn detection), SileroVAD\n    - STT: Google, Azure, OpenAI, AmiVoice\n    - LLM: ChatGPT, OpenAI Responses API (REST / WebSocket), Gemini, Claude, and any model supported by LiteLLM or Dify\n    - TTS: VOICEVOX / AivisSpeech, OpenAI, SpeechGateway (including Style-Bert-VITS2 and Aivis Cloud API)\n- **⚡️ AI Agent native**: Designed to support agentic systems. In addition to standard tool calls, it offers Dynamic Tool Calls for extensibility and supports progress feedback for high-latency operations.\n\n\n## 🚀 Quick start\n\n**Requirements**: Python 3.11+, OpenAI API key, and a running VOICEVOX instance for TTS\n\n### 📺 Local (Console)\n\nInstall AIAvatarKit.\n\n```sh\npip install aiavatar\n```\n\n**NOTE:** If the steps in technical blogs don’t work as expected, the blog may be based on a version prior to v0.6. Some features may be limited, but you can try downgrading with `pip install aiavatar==0.5.8` to match the environment described in the blog.\n\n\nMake the script as `run.py`.\n\n```python\nimport asyncio\nfrom aiavatar import AIAvatar\n\naiavatar_app = AIAvatar(\n    openai_api_key=OPENAI_API_KEY,\n    debug=True\n)\nasyncio.run(aiavatar_app.start_listening())\n```\n\nStart AIAvatar. Also, don't forget to launch VOICEVOX beforehand.\n\n```bash\n$ python run.py\n```\n\nConversation will start when you say the wake word \"こんにちは\" (or \"Hello\" when language is not `ja-JP`).\n\nFeel free to enjoy the conversation afterwards!\n\n\n### 🌐 WebSocket (Browser)\n\nInstall AIAvatarKit and additional dependencies.\n\n```sh\npip install aiavatar fastapi uvicorn websockets\n```\n\nMake the script as `ws.py`.\n\n```python\nfrom fastapi import FastAPI\nfrom fastapi.staticfiles import StaticFiles\nfrom aiavatar.adapter.websocket.server import AIAvatarWebSocketServer\nfrom aiavatar.util import download_example\n\n# Download example UI if not exists\ndownload_example(\"websocket/html\")\n\n# Build Speech-to-Speech pipeline with WebSocket adapter\naiavatar_app = AIAvatarWebSocketServer(\n    openai_api_key=OPENAI_API_KEY\n)\n\n# Build websocket server\napp = FastAPI()\nrouter = aiavatar_app.get_websocket_router()\napp.include_router(router)\napp.mount(\"/static\", StaticFiles(directory=\"html\"), name=\"static\")\n\n# Setup admin panel (Optional)\nfrom aiavatar.admin import setup_admin_panel\nsetup_admin_panel(app, adapter=aiavatar_app)\n```\n\nStart server. Also, don't forget to launch VOICEVOX beforehand.\n\n```bash\n$ python -m uvicorn ws:app\n```\n\nOpen following URLs and enjoy the conversation!\n\n- Character icon (dynamic expression, lip sync, blinking): http://127.0.0.1:8000/static/index.html\n- MotionPNGTuber: http://127.0.0.1:8000/static/mpt.html\n\nYou can also access the Admin Panel at http://127.0.0.1:8000/admin.\n\n\n## 🔖 Contents\n\n- [🎓 Generative AI](#-generative-ai)\n    - [ChatGPT](#chatgpt)\n    - [OpenAI Responses API](#openai-responses-api)\n    - [Claude](#claude)\n    - [Gemini](#gemini)\n    - [Dify](#dify)\n    - [OpenAI-compatible APIs](#openai-compatible-apis)\n    - [Other LLMs](#other-llms)\n\n- [🗣️ Voice](#️voice)\n\n- [👂 Speech Listener](#-speech-listener)\n    - [Preprocessing and Postprocessing](#preprocessing-and-postprocessing)\n    - [Speaker Diarization](#speaker-diarization)\n\n- [🎙️ Speech Detector](#%EF%B8%8F-speech-detector)\n    - [Silero VAD Speech Detector](#silero-speech-detector)\n    - [Silero Stream Speech Detector](#silero-stream-speech-detector)\n    - [Azure Stream Speech Detector](#azure-stream-speech-detector)\n    - [AWS Stream Speech Detector](#aws-stream-speech-detector)\n    - [Customization](#customization)\n    - [Standard Speech Detector (Legacy)](#standard-speech-detector-legacy)\n\n- [🥰 Face Expression](#-face-expression)\n\n- [💃 Animation](#-animation)\n\n- [🥳 Character Management](#-character-management)\n    - [Get started](#get-started)\n    - [Updating Diaries](#updating-diaries)\n    - [Updating Schedules](#updating-schedules)\n    - [Automated Daily Updates](#automated-daily-updates)\n    - [Batch Generation](#batch-generation)\n    - [Long-term Memory](#long-term-memory)\n    - [Binding to Adapter](#binding-to-adapter)\n\n- [🧩 API](#-api)\n    - [💫 RESTful API (SSE)](#-restful-api-sse)\n    - [🔵 Dify-compatible API](#-dify-compatible-api)\n    - [🔌 WebSocket](#-websocket)\n    - [🟩 LINE Bot](#-line-bot)\n\n- [🦜 AI Agent](#-ai-agent)\n    - [⚡️ Tool Call](#️-tool-call)\n    - [⌛️ Tool Call with Streaming Progress](#%EF%B8%8F-tool-call-with-streaming-progress)\n    - [📋 Tool Response Formatter (Direct Response)](#-tool-response-formatter-direct-response)\n    - [🪄 Dynamic Tool Call](#-dynamic-tool-call)\n    - [🔌 MCP](#-mcp)\n    - [🛠️ Built-in Tools](#️-built-in-tools)\n    - [🦞 OpenClaw](#-openclaw)\n\n- [🛡️ Guardrail](#%EF%B8%8F-guardrail)\n\n- [🌎 Platform Guide](#-platform-guide)\n    - [🐈 VRChat](#-vrchat)\n    - [🍓 Raspberry Pi](#-raspberry-pi)\n\n- [⚙️ Administration](#️-administration)\n    - [Admin Panel](#admin-panel)\n    - [REST API](#rest-api)\n    - [📈 Observability](#-observability)\n\n- [🧪 Evaluation](#-evaluation)\n\n- [🤿 Deep Dive](#-deep-dive)\n    - [🐘 PostgreSQL](#-postgresql)\n    - [👀 Vision](#-vision)\n    - [💾 Long-term Memory](#-long-term-memory)\n    - [🐓 Wakeword](#-wakeword)\n    - [📋 System Prompt Parameters](#-system-prompt-parameters)\n    - [🎛️ Inline LLM Parameters](#️-inline-llm-parameters)\n    - [⏰ Timestamp Insertion](#-timestamp-insertion)\n    - [🧵 Request merging](#-request-merging)\n    - [📥 Invoke Queue](#-invoke-queue)\n    - [🧺 Shared Context](#-shared-context)\n    - [🔈 Audio Device](#-audio-device)\n    - [🐆 Quick Response](#-quick-response)\n    - [🎭 Custom Behavior](#-custom-behavior)\n    - [✅ Request Validation](#-request-validation)\n    - [🎚️ Noise Filter](#%EF%B8%8F-noise-filter)\n    - [🔄 Migration Guide: From v0.6.x to v0.7.0](#-migration-guide-from-v06x-to-v070)\n\n\n## 🎓 Generative AI\n\nYou can set model and system prompt when instantiate `AIAvatar`.\n\n```python\naiavatar_app = AIAvatar(\n    openai_api_key=\"YOUR_OPENAI_API_KEY\",\n    openai_model=\"gpt-4o\",\n    system_prompt=\"You are my cat.\"\n)\n```\n\n### ChatGPT\n\nIf you want to configure in detail, create instance of `ChatGPTService` with custom parameters and set it to `AIAvatar`.\n\n```python\n# Create ChatGPTService\nfrom aiavatar.sts.llm.chatgpt import ChatGPTService\nllm = ChatGPTService(\n    openai_api_key=OPENAI_API_KEY,\n    model=\"gpt-4o\",\n    temperature=0.0,\n    system_prompt=\"You are my cat.\"\n)\n\n# Create AIAvatar with ChatGPTService\naiavatar_app = AIAvatar(\n    llm=llm,\n    openai_api_key=OPENAI_API_KEY   # API Key for STT\n)\n```\n\n### OpenAI Responses API\n\nUse `OpenAIResponsesService` to leverage the OpenAI Responses API. Conversation history is managed server-side via `previous_response_id`, eliminating the need for client-side context management.\n\n```python\nfrom aiavatar.sts.llm.openai_responses import OpenAIResponsesService\nllm = OpenAIResponsesService(\n    openai_api_key=OPENAI_API_KEY,\n    model=\"gpt-5.4\",\n    system_prompt=\"You are my cat.\"\n)\n\naiavatar_app = AIAvatar(\n    llm=llm,\n    openai_api_key=OPENAI_API_KEY   # API Key for STT\n)\n```\n\nFor lower latency, use the WebSocket variant. This maintains persistent connections via a connection pool, which can reduce latency by up to 40%, especially in tool-call-heavy workflows.\n\n```python\n# pip install websockets\nfrom aiavatar.sts.llm.openai_responses_websocket import OpenAIResponsesWebSocketService\nllm = OpenAIResponsesWebSocketService(\n    openai_api_key=OPENAI_API_KEY,\n    model=\"gpt-5.4\",\n    reasoning_effort=\"low\",\n    system_prompt=\"You are my cat.\"\n)\n```\n\nNOTE: The WebSocket variant does not support the `temperature` parameter. Use `reasoning_effort` (\"none\", \"low\", \"medium\", \"high\") instead to control response behavior. Dynamic Tool Calls are not supported in either variant, as the server-side history management via `previous_response_id` is incompatible with the pre-flight tool filtering calls.\n\n\n### Claude\n\nCreate instance of `ClaudeService` with custom parameters and set it to `AIAvatar`. The default model is `claude-sonnet-4-5`.\n\n```python\n# Create ClaudeService\nfrom aiavatar.sts.llm.claude import ClaudeService\nllm = ClaudeService(\n    anthropic_api_key=ANTHROPIC_API_KEY,\n    model=\"claude-sonnet-4-5\",\n    temperature=0.0,\n    system_prompt=\"You are my cat.\"\n)\n\n# Create AIAvatar with ClaudeService\naiavatar_app = AIAvatar(\n    llm=llm,\n    openai_api_key=OPENAI_API_KEY   # API Key for STT\n)\n```\n\nNOTE: We support Claude on Anthropic API, not Amazon Bedrock for now. Use LiteLLM or other API Proxies.\n\n\n### Gemini\n\nCreate instance of `GeminiService` with custom parameters and set it to `AIAvatar`. The default model is `gemini-2.0-flash-exp`.\n\n```python\n# Create GeminiService\n# pip install google-genai\nfrom aiavatar.sts.llm.gemini import GeminiService\nllm = GeminiService(\n    gemini_api_key=GEMINI_API_KEY,\n    model=\"gemini-2.0-pro-latest\",\n    temperature=0.0,\n    system_prompt=\"You are my cat.\"\n)\n\n# Create AIAvatar with GeminiService\naiavatar_app = AIAvatar(\n    llm=llm,\n    openai_api_key=OPENAI_API_KEY   # API Key for STT\n)\n```\n\nNOTE: We support Gemini on Google AI Studio, not Vertex AI for now. Use LiteLLM or other API Proxies.\n\n\n### Dify\n\nYou can use the Dify API instead of a specific LLM's API. This eliminates the need to manage code for tools or RAG locally.\n\n```python\n# Create DifyService\nfrom aiavatar.sts.llm.dify import DifyService\nllm = DifyService(\n    api_key=DIFY_API_KEY,\n    base_url=DIFY_URL,\n    user=\"aiavatarkit_user\",\n    is_agent_mode=True\n)\n\n# Create AIAvatar with DifyService\naiavatar_app = AIAvatar(\n    llm=llm,\n    openai_api_key=OPENAI_API_KEY   # API Key for STT\n)\n```\n\n\n### OpenAI-compatible APIs\n\n`ChatGPTService` supports OpenAI-compatible APIs, such as Grok, Gemini, and Claude.\n\nBy specifying the `model`, `openai_api_key`, and `base_url`, these models can now be used with a non-reasoning configuration out of the box.\n\n```python\n# Grok\nMODEL = \"grok-4-1-fast-non-reasoning\"\nOPENAI_API_KEY = \"YOUR_XAI_API_KEY\"\nBASE_URL = \"https://api.x.ai/v1\"\n\n# Gemini on Google AI Studio\nMODEL = \"gemini-2.5-flash\"\nOPENAI_API_KEY = \"YOUR_GEMINI_API_KEY\"\nBASE_URL = \"https://generativelanguage.googleapis.com/v1beta/openai/\"\n\n# Claude on Anthropic\nLLM_MODEL = \"claude-haiku-4-5\"\nOPENAI_API_KEY = \"YOUR_ANTHROPIC_API_KEY\"\nBASE_URL = \"https://api.anthropic.com/v1/\"\n\n# Configure ChatGPTService\nfrom aiavatar.sts.llm.chatgpt import ChatGPTService\nllm = ChatGPTService(\n    openai_api_key=OPENAI_API_KEY,\n    base_url=BASE_URL,\n    model=MODEL,\n    system_prompt=SYSTEM_PROMPT,\n    # extra_body={\"thinking\": { \"type\": \"disabled\"}},   # Claude\n)\n```\n\n\n### Other LLMs\n\nYou can use other LLMs by using `LiteLLMService` or implementing `LLMService` interface.\n\nSee the details of LiteLLM here: https://github.com/BerriAI/litellm\n\n\n### Voice Text Tag (Think Before Answering)\n\nBy setting `voice_text_tag`, you can have the LLM \"think before answering\" (Chain-of-Thought) while vocalizing only the answer portion. You can specify a single tag or a list of tags.\n\n```python\n# Single tag: vocalize only \u003canswer\u003e content\nllm = ChatGPTService(\n    system_prompt=\"Think within \u003cthink\u003e tags. Write your answer within \u003canswer\u003e tags.\",\n    voice_text_tag=\"answer\"\n)\n\n# Multiple tags: vocalize both \u003cack\u003e and \u003canswer\u003e, skip \u003cthink\u003e\nllm = ChatGPTService(\n    system_prompt=\"Output \u003cack\u003efirst reaction\u003c/ack\u003e\u003cthink\u003ereasoning\u003c/think\u003e\u003canswer\u003efull response\u003c/answer\u003e\",\n    voice_text_tag=[\"ack\", \"answer\"]\n)\n```\n\n\n## 🗣️　Voice\n\nYou can set speaker id and the base url for VOICEVOX server when instantiate `AIAvatar`.\n\n```python\naiavatar_app = AIAvatar(\n    openai_api_key=\"YOUR_OPENAI_API_KEY\",\n    # 46 is Sayo. See http://127.0.0.1:50021/speakers to get all ids for characters\n    voicevox_speaker=46\n)\n```\n\nIf you want to configure in detail, create instance of `VoicevoxSpeechSynthesizer` with custom parameters and set it to `AIAvatar`.\nHere is the example for [AivisSpeech](https://aivis-project.com).\n\n```python\n# Create VoicevoxSpeechSynthesizer with AivisSpeech configurations\nfrom aiavatar.sts.tts.voicevox import VoicevoxSpeechSynthesizer\ntts = VoicevoxSpeechSynthesizer(\n    base_url=\"http://127.0.0.1:10101\",  # Your AivisSpeech API server\n    speaker=\"888753761\"   # Anneli\n)\n\n# Create AIAvatar with VoicevoxSpeechSynthesizer\naiavatar_app = AIAvatar(\n    tts=tts,\n    openai_api_key=OPENAI_API_KEY   # API Key for LLM and STT\n)\n```\n\nYou can also set speech controller that uses alternative Text-to-Speech services. We support Azure, Google, OpenAI and any other TTS services supported by [SpeechGateway](https://github.com/uezo/speech-gateway) such as Style-Bert-VITS2 and Aivis Cloud API.\n\n```python\nfrom aiavatar.sts.tts.azure import AzureSpeechSynthesizer\nfrom aiavatar.sts.tts.google import GoogleSpeechSynthesizer\nfrom aiavatar.sts.tts.openai import OpenAISpeechSynthesizer\nfrom aiavatar.sts.tts.speech_gateway import SpeechGatewaySpeechSynthesizer\n```\n\n### Instant TTS Synthesizer\n\nFor quick setup of custom TTS services with HTTP API endpoints, use `create_instant_synthesizer`. This allows you to create a TTS synthesizer with just HTTP request parameters.\n\nExamples:\n\n```python\nfrom aiavatar.sts.tts import create_instant_synthesizer\n\n# Style-Bert-VITS2 API\nsbv2_tts = create_instant_synthesizer(\n    method=\"POST\",\n    url=\"http://127.0.0.1:5000/voice\",\n    json={\n        \"model_id\": \"0\",\n        \"speaker_id\": \"0\",\n        \"text\": \"{text}\"  # Placeholder for processed text\n    }\n)\n\n# ElevenLabs\nelevenlabs_tts = create_instant_synthesizer(\n    method=\"POST\",\n    url=f\"https://api.elevenlabs.io/v1/text-to-speech/{voice_id}\",\n    headers={\n        \"xi-api-key\": ELEVENLABS_API_KEY\n    },\n    json={\n        \"text\": \"{text}\",\n        \"model_id\": \"eleven_v3\",\n        \"output_format\": \"pcm_16000\"\n    }\n)\n\n# Aivis Cloud API\nfrom aiavatar.sts.tts import AudioConverter\naivis_tts = create_instant_synthesizer(\n    method=\"POST\",\n    url=\"https://api.aivis-project.com/v1/tts/synthesize\",\n    headers={\n        \"Content-Type\": \"application/json\",\n        \"Authorization\": f\"Bearer {AIVIS_API_KEY}\"\n    },\n    json={\n        \"model_uuid\": \"22e8ed77-94fe-4ef2-871f-a86f94e9a579\",   # Kohaku\n        \"text\": \"{text}\"\n    },\n    response_parser=AudioConverter(debug=True).convert\n)\n\n# Kotodama API (Implement `make_request` to apply style or language.)\nimport base64\nasync def base64_to_bytes(http_response) -\u003e bytes:\n    response_json = http_response.json()\n    b64audio = response_json[\"audios\"][0]\n    return base64.b64decode(b64audio)\n\nkotodama_tts = create_instant_synthesizer(\n    method=\"POST\",\n    url=f\"https://tts3.spiral-ai-app.com/api/tts_generate\",\n    headers={\n        \"Content-Type\": \"application/json\",\n        \"X-API-Key\": KOTODAMA_API_KEY\n    },\n    json={\n        \"text\": \"{text}\",\n        \"speaker_id\": \"Marlo\",\n        \"decoration_id\": \"neutral\",\n        \"audio_format\": \"wav\"\n    },\n    response_parser=base64_to_bytes\n)\n\n# Coefont\nimport hmac\nimport hashlib\n\ndef make_coefont_request(text: str, style_info: dict, language: str):\n    date = str(int(datetime.now(tz=timezone.utc).timestamp()))\n\n    data = json.dumps({\n        \"coefont\": \"33e0a2ff-5050-434c-9506-defe97e52f15\",  # Yuko Goto\n        \"text\": text\n    })\n\n    signature = hmac.new(\n        key=bytes(COEFONT_ACCESS_SECRET, \"utf-8\"),\n        msg=(date+data).encode(\"utf-8\"),\n        digestmod=hashlib.sha256\n    ).hexdigest()\n\n    return httpx.Request(\n        method=\"post\",\n        url=\"https://api.coefont.cloud/v2/text2speech\",\n        headers={\n            \"Content-Type\": \"application/json\",\n            \"Authorization\": COEFONT_ACCESS_KEY,\n            \"X-Coefont-Date\": date,\n            \"X-Coefont-Content\": signature\n        },\n        data=data\n    )\n\ntts = create_instant_synthesizer(\n    request_maker=make_coefont_request,\n    follow_redirects=True\n)\n\n# Amazon Polly (AWS)\nimport boto3\nfrom botocore.auth import SigV4Auth\nfrom botocore.awsrequest import AWSRequest\n\nregion = \"ap-northeast-1\"\nvoice_id = \"Mizuki\"\n\nsession = boto3.Session()\n# Set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as environment variables\ncredentials = session.get_credentials().get_frozen_credentials()\n\nconvert_pcm_to_wave = AudioConverter(input_sample_rate=16000).pcm_to_wave\n\ndef aws_polly_request_maker(text, style_info=None, language=None):\n    url = f\"https://polly.{region}.amazonaws.com/v1/speech\"\n    body = json.dumps({\n        \"OutputFormat\": \"pcm\",\n        \"SampleRate\": \"16000\",\n        \"Text\": text,\n        \"VoiceId\": voice_id,\n    })\n    aws_request = AWSRequest(method=\"POST\", url=url, data=body, headers={\"Content-Type\": \"application/json\"})\n    SigV4Auth(credentials, \"polly\", region).add_auth(aws_request)\n    return httpx.Request(method=\"POST\", url=url, headers=dict(aws_request.headers), content=body)\n\ntts = create_instant_synthesizer(\n    request_maker=aws_polly_request_maker,\n    response_parser=convert_pcm_to_wave,\n)\n```\n\nThe `{text}` and `{language}` placeholders in params, headers, and json will be automatically replaced with the processed text and language values during synthesis.\n\n\nYou can also make custom tts components by impelemting `SpeechSynthesizer` interface.\n\n### TTS Caching\n\nAll TTS synthesizers support optional response caching. When `cache_dir` is set, synthesized audio is saved to disk and reused for identical requests, avoiding redundant API calls.\n\n```python\ntts = AzureSpeechSynthesizer(\n    azure_api_key=AZURE_API_KEY,\n    azure_region=AZURE_REGION,\n    speaker=\"ja-JP-MayuNeural\",\n    cache_dir=\"./tts_cache/azure\",  # Enable caching\n    cache_ext=\"wav\",                # File extension (default: \"wav\")\n)\n```\n\n- Cache files are stored as `{sha256_hash}.{cache_ext}` in the specified directory\n- The hash is computed from all request parameters (URL, headers, body, etc.)\n- Set `cache_dir=None` (default) to disable caching\n- Works with all TTS classes: Azure, OpenAI, Google, Voicevox, and InstantSynthesizer\n- `SpeechGatewaySpeechSynthesizer` does not use this cache as it caches on the gateway side\n\n### Preprocessing\n\nAIAvatarKit provides text preprocessing functionality that transforms text before Text-to-Speech processing. This enables improved speech quality and conversion of specific text patterns.\n\n#### Alphabet to Katakana Conversion\n\nA preprocessor that converts alphabet text to katakana using LLM. Supports kana_map for storing word-to-reading mappings to reduce latency on repeated words.\n\n```python\nfrom aiavatar.sts.tts.preprocessor.alphabet2kana import AlphabetToKanaPreprocessor\n\n# Create preprocessor with kana_map for pre-registered word-reading mappings\nalphabet2kana_preproc = AlphabetToKanaPreprocessor(\n    openai_api_key=OPENAI_API_KEY,\n    model=\"gpt-4o-mini\",                      # Model to use (default: gpt-4.1-mini)\n    alphabet_length=3,                        # Minimum alphabet length to convert (default: 3)\n    special_chars=\".'-'−–\",                   # Characters that connect words (default: \".'-'−–\")\n    use_kana_map=True,                        # Enable kana_map mode (default: True)\n    kana_map={\"GitHub\": \"ギットハブ\"},         # Pre-registered word-reading mappings (optional)\n    debug=True,                               # Enable debug logging (default: False)\n)\n\n# Add to TTS\ntts.preprocessors.append(alphabet2kana_preproc)\n\n# Words converted by LLM are automatically added to kana_map\n# You can persist and restore kana_map for future sessions:\nimport json\n# Save\nwith open(\"kana_map.json\", \"w\") as f:\n    json.dump(alphabet2kana_preproc.kana_map, f, ensure_ascii=False)\n# Load\nwith open(\"kana_map.json\") as f:\n    kana_map = json.load(f)\n```\n\nKey features:\n- **kana_map**: Pre-register known word-reading mappings and automatically add LLM results to avoid repeated API calls\n- **special_chars**: Words containing these characters (e.g., `Mr.`, `You're`, `Wi-Fi`) are always processed regardless of `alphabet_length`\n- **Case-insensitive**: Matches `API`, `api`, and `Api` with a single kana_map entry\n- **debug mode**: Logs `[KanaMap]` for cached hits and `[LLM]` for new readings with elapsed time\n\n#### Pattern Match Conversion\n\nYou can also use regular expressions and string patterns for conversion:\n\n```python\nfrom aiavatar.sts.tts.preprocessor.patternmatch import PatternMatchPreprocessor\n\n# Create pattern match preprocessor\npattern_preproc = PatternMatchPreprocessor(patterns=[\n    (\"API\", \"エーピーアイ\"),               # Fixed string replacement\n    (\"URL\", \"ユーアールエル\"),\n    (r\"\\d+\", lambda m: \"number\"),          # Regex replacement with function\n])\n\n# Add common patterns\npattern_preproc.add_number_dash_pattern()  # Number-dash patterns (e.g., 12-34 → イチニの サンヨン)\npattern_preproc.add_phonenumber_pattern()  # Phone number patterns\n\n# Add to TTS\ntts.preprocessors.append(pattern_preproc)\n```\n\n#### Creating Custom Preprocessors\n\nYou can create your own preprocessors by implementing the `TTSPreprocessor` interface:\n\n```python\nfrom aiavatar.sts.tts.preprocessor import TTSPreprocessor\n\nclass CustomPreprocessor(TTSPreprocessor):\n    def __init__(self, custom_dict: dict = None):\n        self.custom_dict = custom_dict or {}\n    \n    async def process(self, text: str, style_info: dict = None, language: str = None) -\u003e str:\n        # Custom conversion logic\n        processed_text = text\n        \n        # Dictionary-based replacement\n        for original, replacement in self.custom_dict.items():\n            processed_text = processed_text.replace(original, replacement)\n        \n        # Language-specific conversions\n        if language == \"ja-JP\":\n            processed_text = processed_text.replace(\"OK\", \"オーケー\")\n        \n        return processed_text\n\n# Use custom preprocessor\ncustom_preproc = CustomPreprocessor(custom_dict={\n    \"GitHub\": \"ギットハブ\",\n    \"Python\": \"パイソン\",\n    \"Docker\": \"ドッカー\"\n})\n\ntts.preprocessors.append(custom_preproc)\n```\n\n#### Combining Preprocessors\n\nMultiple preprocessors can be used together. They are executed in the order they were registered:\n\n```python\n# Combine multiple preprocessors\ntts.preprocessors.extend([\n    pattern_preproc,        # 1. Pattern match conversion\n    alphabet2kana_preproc,  # 2. Alphabet to katakana conversion\n    custom_preproc          # 3. Custom conversion\n])\n```\n\n\n### Adjusting Speech Speed\n\nWith `SpeechGatewaySpeechSynthesizer`, you can change the speech speed per session by setting the speed either on the entire instance or in `style_info`.\n\nHere is an example of storing the speech speed as `tts_speed` in session data when using WebSocketAdapter.\n\n```python\n# Apply speech speed per session\nfrom aiavatar.sts.llm import LLMResponse\n@aiavatar_app.sts.process_llm_chunk\nasync def process_llm_chunk(llm_stream_chunk: LLMResponse, session_id: str, user_id: str) -\u003e dict:\n    if session_data := aiavatar_app.sessions.get(session_id):\n        if speed := session_data.data.get(\"tts_speed\"):\n            return {\"speed\": float(speed)}\n```\n\nNOTE: To configure `tts_speed`, you can either set up a REST API endpoint to update it directly, or use control tags included in responses to update it.\n\n\n## 👂 Speech listener\n\nIf you want to configure in detail, create instance of `SpeechRecognizer` with custom parameters and set it to `AIAvatar`. We support Azure, Google and OpenAI Speech-to-Text services.\n\nNOTE: **`AzureSpeechRecognizer` is much faster** than Google and OpenAI(default).\n\n```python\n# Create AzureSpeechRecognizer\nfrom aiavatar.sts.stt.azure import AzureSpeechRecognizer\nstt = AzureSpeechRecognizer(\n    azure_api_key=AZURE_API_KEY,\n    azure_region=AZURE_REGION\n)\n\n# Create AIAvatar with AzureSpeechRecognizer\naiavatar_app = AIAvatar(\n    stt=stt,\n    openai_api_key=OPENAI_API_KEY   # API Key for LLM\n)\n```\n\nYou can also make custom STT components by implementing `SpeechRecognizer` interface.\n\n### Preprocessing and Postprocessing\n\nYou can add custom preprocessing and postprocessing to any `SpeechRecognizer` implementation. This is useful for tasks like speaker verification, audio filtering, or text normalization.\n\n```python\nfrom aiavatar.sts.stt.openai import OpenAISpeechRecognizer\n\n# Create recognizer\nrecognizer = OpenAISpeechRecognizer(openai_api_key=\"your-api-key\")\n\n# Add preprocessing - e.g., speaker verification\n@recognizer.preprocess\nasync def verify_speaker(session_id: str, audio_data: bytes):\n    # Perform speaker verification\n    is_valid_speaker = await check_speaker_identity(audio_data)\n    \n    if not is_valid_speaker:\n        # Return empty bytes to skip transcription\n        return b\"\", {\"rejected\": True, \"reason\": \"speaker_mismatch\"}\n    \n    # Return processed audio and metadata\n    filtered_audio = apply_noise_filter(audio_data)\n    return filtered_audio, {\"speaker_verified\": True, \"session_id\": session_id}\n\n# Add postprocessing - e.g., text formatting\n@recognizer.postprocess\nasync def format_text(session_id: str, text: str, audio_data: bytes, preprocess_metadata: dict):\n    # Format transcribed text\n    formatted_text = text.strip().capitalize()\n    \n    # Add punctuation if missing\n    if formatted_text and formatted_text[-1] not in '.!?':\n        formatted_text += '.'\n    \n    # Return formatted text and metadata\n    return formatted_text, {\n        \"original_text\": text,\n        \"formatting_applied\": True,\n        \"preprocess_info\": preprocess_metadata\n    }\n\n# Use the recognizer with preprocessing and postprocessing\nresult = await recognizer.recognize(\n    session_id=\"user-123\",\n    data=audio_bytes\n)\n\nprint(f\"Text: {result.text}\")\nprint(f\"Preprocess metadata: {result.preprocess_metadata}\")\nprint(f\"Postprocess metadata: {result.postprocess_metadata}\")\n```\n\nThe preprocessing and postprocessing functions can return either:\n- Just the processed data (bytes for preprocess, string for postprocess)\n- A tuple of (processed_data, metadata_dict) for additional information\n\nIf preprocessing returns empty bytes, the transcription is skipped and the result will have `text=None`.\n\n\n### Speaker Diarization\n\nAIAvatarKit provides speaker diarization functionality to suppress responses to voices other than the main speaker. This prevents interruptions from surrounding conversations or venue announcements at events.\n\nThe `MainSpeakerGate` provides the following features:\n\n- Calculates voice embeddings from request audio\n- Registers a voice as the main speaker when similarity exceeds threshold for 2 consecutive requests (per session)\n- Returns `accepted=True` when request audio similarity exceeds threshold after main speaker registration\n- Returns `accepted=True` when no main speaker is registered yet\n\n**NOTE:** While mechanically ignoring non-main speaker voices (Example 1) is simplest, it risks stopping conversation due to misidentification and cannot handle speaker changes. Consider context-aware handling (Example 2) as well.\n\n```python\nfrom aiavatar.sts.stt.speaker_gate import MainSpeakerGate\nspeaker_gate = MainSpeakerGate()\n\n# Example 1: Drop request when the voice is not from main speaker\n@aiavatar_app.sts.stt.preprocess\nasync def stt_preprocess(session_id: str, audio_bytes: bytes):\n    # Compare with main speaker's voice embedding\n    gate_response = await speaker_gate.evaluate(session_id, audio_bytes, aiavatar_app.sts.vad.sample_rate)\n    # Branch processing based on similarity with main speaker's voice\n    if not gate_response.accepted:\n        logger.info(f\"Ignore other speaker's voice: confidence={gate_response.confidence}\")\n        return None, gate_response.to_dict()\n    else:\n        return audio_bytes, gate_response.to_dict()\n\n# Example 2: Add annotation for LLM that the voice is not from main speaker\n@aiavatar_app.sts.stt.postprocess\nasync def stt_postprocess(session_id: str, text: str, audio_bytes: bytes, preprocess_metadata: dict):\n    # Compare with main speaker's voice embedding\n    gate_response = await speaker_gate.evaluate(session_id, audio_bytes, aiavatar_app.sts.vad.sample_rate)\n    # Branch processing based on similarity with main speaker's voice\n    if not gate_response.accepted:\n        logger.info(f\"Adding note that this may be from a different speaker: confidence={gate_response.confidence}\")\n        return f\"$The following request may not be from the main speaker (similarity: {gate_response.confidence}). Determine from the content whether to respond. If you should not respond, output just[wait:user] as the answer:\\n\\n{text}\", gate_response.to_dict()\n    else:\n        return text, gate_response.to_dict()\n```\n\n\n## 🎙️ Speech Detector\n\nAIAvatarKit includes Voice Activity Detection (VAD) components to automatically detect when speech starts and ends in audio streams. This enables seamless conversation flow without manual input controls.\n\n### Silero Speech Detector\n\nThe default Speech Detector is `SileroSpeechDetector`, which employs AI-based voice activity detection using the Silero VAD model:\n\n```python\nfrom aiavatar.sts.vad.silero import SileroSpeechDetector\n\nvad = SileroSpeechDetector(\n    speech_probability_threshold=0.5,    # AI model confidence threshold (0.0-1.0)\n    silence_duration_threshold=0.5,      # Seconds of silence to end recording\n    volume_db_threshold=None,            # Optional: filter by volume in dB (e.g., -30.0)\n    max_duration=10.0,                   # Maximum recording duration\n    min_duration=0.2,                    # Minimum recording duration\n    sample_rate=16000,                   # Audio sample rate\n    channels=1,                          # Audio channels\n    chunk_size=512,                      # Audio processing chunk size\n    model_pool_size=1,                   # Number of parallel AI models\n    debug=True\n)\n\naiavatar_app = AIAvatar(vad=vad, openai_api_key=OPENAI_API_KEY)\n```\n\nFor high-concurrency applications:\n\n```python\nvad = SileroSpeechDetector(\n    speech_probability_threshold=0.6,    # Stricter threshold for noisy environments\n    model_pool_size=4,                   # 4 parallel AI models for load balancing\n    debug=False\n)\n```\n\n\n### Silero Stream Speech Detector\n\n`SileroStreamSpeechDetector` extends `SileroSpeechDetector` with segment-based speech recognition. It performs partial transcription during recording, allowing you to receive intermediate results before the final transcription.\n\n```python\nfrom aiavatar.sts.vad.stream import SileroStreamSpeechDetector\nfrom aiavatar.sts.stt.google import GoogleSpeechRecognizer\n\nvad = SileroStreamSpeechDetector(\n    speech_recognizer=GoogleSpeechRecognizer(...),\n    segment_silence_threshold=0.2,       # Silence duration to trigger segment recognition\n    silence_duration_threshold=0.5,      # Silence duration to finalize recording\n    # Inherits all SileroSpeechDetector parameters\n)\n```\n\n#### Segment Recognition Callback\n\nThe `on_speech_detecting` callback is triggered when a speech segment is recognized:\n\n```python\n@vad.on_speech_detecting\nasync def on_speech_detecting(text, session):\n    print(f\"Partial text: {text}\")\n\n    # For WebSocket apps, send partial text to client via info message\n    # resp = STSResponse(\n    #     type=\"info\",\n    #     session_id=session.session_id,\n    #     metadata={\"partial_request_text\": text}\n    # )\n    # await ws_app.handle_response(resp)\n```\n\n#### Text Validation\n\nUse `validate_recognized_text` to filter out invalid recognition results:\n\n```python\n@vad.validate_recognized_text\ndef validate(text):\n    if len(text) \u003c 2:\n        return \"Text too short\"  # Return error message to reject\n    return None  # Return None to accept\n```\n\n\n### Azure Stream Speech Detector\n\n`AzureStreamSpeechDetector` uses Azure's streaming speech recognition service for both speech detection and transcription. Audio is continuously streamed to Azure, and speech boundaries are determined by Azure's recognition events.\n\n```sh\npip install azure-cognitiveservices-speech\n```\n\n```python\nfrom aiavatar.sts.vad.azure_stream import AzureStreamSpeechDetector\n\nvad = AzureStreamSpeechDetector(\n    azure_subscription_key=AZURE_API_KEY,\n    azure_region=AZURE_REGION\n)\n```\n\nThis detector also supports the `on_speech_detecting` callback for partial transcription results:\n\n```python\n@vad.on_speech_detecting\nasync def on_speech_detecting(text, session):\n    print(f\"Partial text: {text}\")\n\n    # For WebSocket apps, send partial text to client via info message\n    # resp = STSResponse(\n    #     type=\"info\",\n    #     session_id=session.session_id,\n    #     metadata={\"partial_request_text\": text}\n    # )\n    # await ws_app.handle_response(resp)\n```\n\n### AWS Stream Speech Detector\n\n`AmazonTranscribeStreamSpeechDetector` uses Amazon Transcribe's streaming speech recognition service for both speech detection and transcription. Audio is continuously streamed to Amazon Transcribe, and speech boundaries are determined by the recognition results combined with a configurable silence duration threshold.\n\n```sh\npip install amazon-transcribe\n```\n\n```python\nfrom aiavatar.sts.vad.amazon_transcribe_stream import AmazonTranscribeStreamSpeechDetector\n\nvad = AmazonTranscribeStreamSpeechDetector(\n    aws_region=\"ap-northeast-1\",\n    aws_access_key_id=AWS_ACCESS_KEY_ID,         # Optional: uses default credential chain if omitted\n    aws_secret_access_key=AWS_SECRET_ACCESS_KEY,  # Optional: uses default credential chain if omitted\n    aws_language=\"ja-JP\",\n    silence_duration_threshold=0.5,  # Seconds of silence after last recognition to finalize\n    max_duration=20.0,               # Maximum recording duration in seconds\n)\n```\n\nWhen `silence_duration_threshold \u003e 0`, multiple recognition results from Amazon Transcribe are accumulated into a single speech detection event. A silence timer starts after each final result, and if new speech arrives before the timer expires, the timer is cancelled and transcription continues. This allows natural pauses within a sentence without splitting the utterance.\n\n\u003e **Note:** The `silence_duration_threshold` timer starts from when Amazon Transcribe returns a final recognition result, not from when the user actually stops speaking. Since Amazon Transcribe takes some time to process audio and return a final result, the actual delay from the user's perspective is: **Transcribe processing delay + `silence_duration_threshold`**. For example, if Transcribe takes ~0.5s to return a final result and `silence_duration_threshold=0.5`, the total delay from the end of speech to firing `on_speech_detected` will be approximately 1.0s.\n\nWhen `max_duration` is reached during recording, if there are accumulated recognition results, speech detection is triggered immediately with the combined text.\n\nThis detector also supports the `on_speech_detecting` callback for partial transcription results. When texts have been accumulated from previous final results, they are prepended to the current partial text:\n\n```python\n@vad.on_speech_detecting\nasync def on_speech_detecting(text, session):\n    print(f\"Partial text: {text}\")\n\n    # For WebSocket apps, send partial text to client via info message\n    # resp = STSResponse(\n    #     type=\"info\",\n    #     session_id=session.session_id,\n    #     metadata={\"partial_request_text\": text}\n    # )\n    # await ws_app.handle_response(resp)\n```\n\nUse `validate_recognized_text` to filter out invalid recognition results:\n\n```python\n@vad.validate_recognized_text\ndef validate(text):\n    if len(text) \u003c 2:\n        return \"Text too short\"  # Return error message to reject\n    return None  # Return None to accept\n```\n\n\n### Customization\n\n#### on_recording_started Callback\n\nThe `on_recording_started` callback is triggered when recording has been active long enough to be considered meaningful speech. This is useful for stopping AI speech when the user starts talking.\n\n```python\n# Option 1: Pass callback in constructor\nasync def my_recording_started_handler(session_id: str):\n    print(f\"Recording started for session: {session_id}\")\n    await stop_ai_speech()\n\nvad = SileroSpeechDetector(\n    on_recording_started=my_recording_started_handler,\n    on_recording_started_min_duration=1.5,    # Trigger after 1.5 sec of speech (default)\n    # other parameters...\n)\n\n# Option 2: Use decorator\n@vad.on_recording_started\nasync def on_recording_started(session_id):\n    await stop_ai_speech()\n```\n\nFor stream-based detectors (`SileroStreamSpeechDetector`, `AzureStreamSpeechDetector`), the callback can also be triggered by recognized text length:\n\n```python\nvad = SileroStreamSpeechDetector(\n    speech_recognizer=speech_recognizer,\n    on_recording_started_min_duration=1.5,    # Trigger after 1.5 sec of speech\n    on_recording_started_min_text_length=2,   # OR trigger when text \u003e= 2 chars\n)\n```\n\n#### Custom Trigger Condition\n\nYou can customize when `on_recording_started` fires using the `should_trigger_recording_started` decorator:\n\n```python\n@vad.should_trigger_recording_started\ndef custom_trigger(text, session):\n    # text: Recognized text (None for non-stream detectors)\n    # session: Recording session object\n    # Return True to trigger the callback\n    return text and len(text) \u003e= 5\n```\n\n\n### Standard Speech Detector (Legacy)\n\n`StandardSpeechDetector` uses simple volume-based detection. Consider using `SileroSpeechDetector` for better accuracy. This detector is suitable for environments with limited computing resources:\n\n```python\nfrom aiavatar.sts.vad.standard import StandardSpeechDetector\n\nvad = StandardSpeechDetector(\n    volume_db_threshold=-30.0,           # Voice detection threshold in dB\n    silence_duration_threshold=0.5,      # Seconds of silence to end recording\n    max_duration=10.0,                   # Maximum recording duration\n    min_duration=0.2,                    # Minimum recording duration\n    sample_rate=16000,                   # Audio sample rate\n    channels=1,                          # Audio channels\n    preroll_buffer_count=5,              # Pre-recording buffer size\n    debug=True\n)\n```\n\n\n## 🥰 Face expression\n\nTo control facial expressions within conversations, set the facial expression names and values in `FaceController.faces` as shown below, and then include these expression keys in the response message by adding instructions to the prompt.\n\n```python\naiavatar_app.face_controller.faces = {\n    \"neutral\": \"🙂\",\n    \"joy\": \"😀\",\n    \"angry\": \"😠\",\n    \"sorrow\": \"😞\",\n    \"fun\": \"🥳\"\n}\n\naiavatar_app.sts.llm.system_prompt = \"\"\"# Face Expression\n\n* You have the following expressions:\n\n- joy\n- angry\n- sorrow\n- fun\n\n* If you want to express a particular emotion, please insert it at the beginning of the sentence like [face:joy].\n\nExample\n[face:joy]Hey, you can see the ocean! [face:fun]Let's go swimming.\n\"\"\"\n```\n\n\u003e **Note:** XML-style tags are also supported: `\u003cface name=\"joy\" /\u003e`, `\u003canimation name=\"wave_hands\" /\u003e`. Both bracket and XML formats can be used interchangeably.\n\nThis allows emojis like 🥳 to be autonomously displayed in the terminal during conversations. To actually control the avatar's facial expressions in a metaverse platform, instead of displaying emojis like 🥳, you will need to use custom implementations tailored to the integration mechanisms of each platform. Please refer to our `VRChatFaceController` as an example.\n\n\n## 💃 Animation\n\nNow writing... ✍️\n\n\n## 🥳 Character Management\n\n`CharacterService` provides functionality for managing AI character settings and generating dynamic content such as schedules and diaries based on character personalities.\n\nSchedules and diaries are generated as if by the character's own will. By updating these daily and incorporating them into prompts, you can make the character feel like they are actually living in real-world time.\n\n**Note:** This feature requires PostgreSQL as the database backend.\n\n\n### Get started\n\nRegister a new character using a character setting prompt. At this time, both the weekly schedule and today's schedule are also generated.\n\n```python\nfrom datetime import date\nfrom aiavatar.character import CharacterService\n\n# Initialize service\ncharacter_service = CharacterService(\n    openai_api_key=\"YOUR_API_KEY\"\n)\n\n# Initialize a new character with weekly and daily schedules\ncharacter, weekly, daily = await character_service.initialize_character(\n    name=\"Alice\",\n    character_prompt=\"You are Alice, a cheerful high school student who loves reading...\"\n)\n\nprint(f\"Character ID: {character.id}\")\n```\n\nTo use the registered and generated content as a system prompt, implement `LLMService.get_system_prompt` as follows:\n\n```python\n@llm.get_system_prompt\nasync def get_system_prompt(context_id: str, user_id: str, system_prompt_params: dict):\n    return await character_service.get_system_prompt(\n        character_id=\"YOUR_CHARACTER_ID\",\n        system_prompt_params=system_prompt_params\n    )\n```\n\nThis system prompt includes not only the character settings from `character_prompt`, but also the schedule for the day.\n\n\n### Updating Diaries\n\nDiaries can be automatically generated using `create_diary_with_generation`. The following information is used:\n\n- Character settings\n- Today's schedule\n- Today's news (retrieved via web search)\n- Previous day's diary\n\n```python\n# Generate diary from daily activities\ndiary = await character_service.create_diary_with_generation(\n    character_id=character.id,\n    diary_date=date.today()\n)\n```\n\nThe generated diary can be used as context for the LLM using `GetDiaryTool`. By setting `include_schedule=True`, the schedule information for the day is also retrieved (default is `True`).\n\n```python\nfrom aiavatar.character.tools import GetDiaryTool\nllm.add_tool(\n    GetDiaryTool(\n        character_service=character_service,\n        character_id=YOUR_CHARACTER_ID,\n        include_schedule=True\n    )\n)\n```\n\n\n### Updating Schedules\n\nDaily schedules can be automatically generated using `create_daily_schedule_with_generation`. The following information is used:\n\n- Character settings\n- Weekly schedule\n- Previous day's schedule\n\n```python\ndaily_schedule = await character_service.create_daily_schedule_with_generation(\n    character_id=character.id,\n    schedule_date=date.today()\n)\n```\n\n### Automated Daily Updates\n\nFor a more realistic character experience, use a scheduler service (such as cron) to automatically update schedules and diaries:\n\n- **Daily schedule**: Generate at the beginning of each day (e.g., 0:00 or 6:00)\n- **Diary**: Generate at the end of each day (e.g., 23:00)\n\nExample cron configuration:\n\n```\n# Generate daily schedule at 6:00 AM\n0 6 * * * /usr/bin/python3 /path/to/generate_schedule.py\n\n# Generate diary at 11:00 PM\n0 23 * * * /usr/bin/python3 /path/to/generate_diary.py\n```\n\nExample script for `generate_schedule.py`:\n\n```python\nimport asyncio\nfrom datetime import date\nfrom aiavatar.character import CharacterService\n\nasync def main():\n    character_service = CharacterService(\n        openai_api_key=\"YOUR_API_KEY\"\n    )\n    await character_service.create_daily_schedule_with_generation(\n        character_id=\"YOUR_CHARACTER_ID\",\n        schedule_date=date.today()\n    )\n\nasyncio.run(main())\n```\n\n### Batch Generation\n\nYou can batch generate daily schedules and diaries for a date range using `create_activity_range_with_generation`.\n\n```python\nawait character_service.create_activity_range_with_generation(\n    character_id=YOUR_CHARACTER_ID,\n    start_date=date(2026, 1, 8),\n    end_date=date(2026, 1, 16),  # Defaults to today if omitted\n    overwrite=False,\n)\n```\n\nThis is useful for recovering data when automatic updates were stopped, or for building up initial data when creating a new character.\n\n### Long-term Memory\n\nThis feature is **optional**. If you want to make diaries searchable as long-term memory, you can integrate with an external memory service by configuring `MemoryClient`:\n\n```python\nfrom aiavatar.character import CharacterService, MemoryClient\n\nmemory_client = MemoryClient(base_url=\"http://memory-service:8000\")\n\ncharacter_service = CharacterService(\n    openai_api_key=\"YOUR_API_KEY\",\n    memory_client=memory_client\n)\n```\n\nRegistered diaries can be included in search results using the `search` method.\n\n```python\n# In addition to diaries, conversation history with users and other knowledge are searched comprehensively\nresult = await character_service.memory.search(\n    character_id=\"YOUR_CHARACTER_ID\",\n    user_id=\"YOUR_USER_ID\",\n    query=\"travel summer 2026\"\n)\n```\n\nThe default `MemoryClient` uses [ChatMemory](https://github.com/uezo/chatmemory) as its backend, but you can also use other long-term memory services by inheriting from `MemoryClientBase`.\n\n\n### Binding to Adapter\n\nThe `bind_character` function provides a convenient way to integrate character management with your AIAvatar application. It automatically configures the system prompt, user management, and character-related tools in a single call.\n\n```python\nfrom aiavatar.character import CharacterService\nfrom aiavatar.character.binding import bind_character\n\ncharacter_service = CharacterService(\n    openai_api_key=\"YOUR_API_KEY\"\n)\n\nbind_character(\n    adapter=aiavatar_app,\n    character_service=character_service,\n    character_id=\"YOUR_CHARACTER_ID\",\n    default_user_name=\"You\"\n)\n```\n\nThis single function call sets up:\n\n- **System prompt**: Automatically retrieves the character's system prompt with user-specific parameters\n- **User management**: Creates a new user with `default_user_name` if the user doesn't exist\n- **Username sync**: Sends the username and character name to the client on connection, and updates when changed\n- **Tools**: Registers the following tools automatically:\n  - `UpdateUsernameTool`: Allows the character to update the user's name during conversation\n  - `GetDiaryTool`: Retrieves the character's diary and schedule\n  - `MemorySearchTool`: Searches long-term memory (only if `memory_client` is configured)\n\n\n### CharacterLoader (Lightweight Alternative)\n\n`CharacterLoader` is a lightweight alternative to `CharacterService` that loads character settings from local files instead of a database. No database or external API is required — just plain markdown and JSON files.\n\nThis is ideal when you want to quickly set up a character without infrastructure, or when you prefer to manage character definitions as files.\n\n#### Single file mode\n\nThe simplest usage is to point to a single markdown file containing the system prompt:\n\n```python\nfrom aiavatar.character.loader import CharacterLoader\n\nloader = CharacterLoader(\"system_prompt.md\")\n\n# Bind to LLM service\nloader.bind(adapter.sts.llm)\n```\n\n#### Directory mode\n\nFor richer character definitions, use directory mode with `split_initial_messages=True`. Initial messages are prepended to the conversation history as pseudo user/assistant turns, allowing you to inject character knowledge (episodes, attributes, conversation examples) without overloading the system prompt. Point to a directory containing:\n\n```\nmy_character/\n├── character.md                # Character settings (required with split_initial_messages)\n├── response_instructions.md    # Response rules (optional, appended to system prompt)\n├── message_templates.json      # Template definitions for initial messages\n├── episode.md                  # Character's past experiences (optional)\n├── attribute.md                # Likes, dislikes, personality traits (optional)\n└── conversation_example.md     # Example dialogues for tone reference (optional)\n```\n\n```python\nloader = CharacterLoader(\n    \"my_character\",\n    split_initial_messages=True,\n    lang=\"ja\",\n    user_names={\"user_001\": \"Alice\"},\n    default_user_name=\"You\"\n)\n\nloader.bind(adapter.sts.llm)\n```\n\nThe `message_templates.json` defines how initial messages and self-introduction are structured:\n\n```json\n{\n    \"initial_message_defs\": {\n        \"ja\": {\n            \"self_intro\": \"わかりました。{username}さんですね。\",\n            \"episode\": \"わかりました。\",\n            \"attribute\": \"わかりました。\"\n        }\n    },\n    \"prefixes\": {\n        \"ja\": {\n            \"episode\": \"以下はあなたの過去の経験です。\\n\\n\",\n            \"attribute\": \"以下はあなたの属性情報です。\\n\\n\"\n        }\n    },\n    \"self_intro_template\": {\n        \"ja\": \"$ユーザーの名前は{username}です。\"\n    }\n}\n```\n\n#### Hot reload\n\nAll files are cached with mtime-based invalidation. Edit any file while the application is running, and changes will be reflected on the next request — no restart needed.\n\n#### Custom user name resolution\n\nUse the `@loader.get_user_name` decorator to resolve user names dynamically (e.g., from a database or external service):\n\n```python\n@loader.get_user_name\ndef get_user_name(user_id: str):\n    return db.get_username(user_id)\n```\n\n#### Custom message formatting\n\nUse the `@loader.format_messages` decorator to post-process initial messages before they are sent to the LLM:\n\n```python\n@loader.format_messages\ndef format_messages(messages):\n    # Add timestamps, filter messages, etc.\n    return messages\n```\n\n#### Comparison with CharacterService\n\n| | CharacterLoader | CharacterService |\n|---|---|---|\n| Data source | Local files (`.md`, `.json`) | Database (SQLite / PostgreSQL) |\n| Dependencies | None (standard library only) | `openai`, database libraries |\n| Schedule / Diary generation | Not supported | Auto-generated via LLM |\n| Long-term memory | Not supported | Supported via MemoryClient |\n| Character tools | Not included | username update, diary, memory search |\n| Hot reload | Supported (mtime-based) | Not supported |\n\n\n## 🧩 API\n\nYou can host AIAvatarKit on a server to enable multiple clients to have independent context-aware conversations via RESTful API with streaming responses (Server-Sent Events) and WebSocket.\n\n### 💫 RESTful API (SSE)\n\nBelow is the simplest example of a server program:\n\n```python\nfrom fastapi import FastAPI\nfrom aiavatar.adapter.http.server import AIAvatarHttpServer\n\n# AIAvatar\naiavatar_app = AIAvatarHttpServer(\n    openai_api_key=OPENAI_API_KEY,\n    debug=True\n)\n\n# Setup FastAPI app with AIAvatar components \napp = FastAPI()\nrouter = aiavatar_app.get_api_router()\napp.include_router(router)\n```\n\nSave the above code as `server.py` and run it using:\n\n```sh\nuvicorn server:app\n```\n\n\nNext is the simplest example of a client program:\n\n```python\nimport asyncio\nfrom aiavatar.adapter.http.client import AIAvatarHttpClient\n\naiavatar_app = AIAvatarHttpClient(\n    debug=True\n)\nasyncio.run(aiavatar_app.start_listening(session_id=\"http_session\", user_id=\"http_user\"))\n```\n\nSave the above code as `client.py` and run it using:\n\n```sh\npython client.py\n```\n\nYou can now perform voice interactions just like when running locally.\n\n\nWhen using the streaming API via HTTP, clients communicate with the server using JSON-formatted requests.\n\nBelow is the format for initiating a session:\n\n```json\n{\n    \"type\": \"start\",          // Always `start`\n    \"session_id\": \"6d8ba9ac-a515-49be-8bf4-cdef021a169d\",\n    \"user_id\": \"user_id\",\n    \"context_id\": \"c37ac363-5c65-4832-aa25-fd3bbbc1b1e7\",   // Set null or provided id in `start` response\n    \"text\": \"こんにちは\",       // If set, audio_data will be ignored         \n    \"audio_data\": \"XXXX\",     // Base64 encoded audio data\n    \"files\": [\n        {\n            \"type\": \"image\",        // Only `image` is supported for now\n            \"url\": \"https://xxx\",\n        }\n    ],\n    \"metadata\": {}\n}\n```\n\nThe server returns responses as a stream of JSON objects in the following structure.\n\nThe communication flow typically consists of:\n\n```json\n{\n    \"type\": \"chunk\",    // start -\u003e chunk -\u003e final\n    \"session_id\": \"6d8ba9ac-a515-49be-8bf4-cdef021a169d\",\n    \"user_id\": \"user01\",\n    \"context_id\": \"c37ac363-5c65-4832-aa25-fd3bbbc1b1e7\",\n    \"text\": \"[face:joy]こんにちは！\",   // Response text with info\n    \"voice_text\": \"こんにちは！\",       // Response text for voice synthesis\n    \"avatar_control_request\": {\n        \"animation_name\": null,       // Parsed animation name\n        \"animation_duration\": null,   // Parsed duration for animation\n        \"face_name\": \"joy\",           // Parsed facial expression name\n        \"face_duration\": 4.0          // Parsed duration for the facial expression\n    },\n    \"audio_data\": \"XXXX\",   // Base64 encoded. Playback this as the character's voice.\n    \"metadata\": {\n        \"is_first_chunk\": true\n    }\n}\n```\n\n\nYou can test the streaming API using a simple curl command:\n\n```sh\ncurl -N -X POST http://127.0.0.1:8000/chat \\\n    -H \"Content-Type: application/json\" \\\n    -d '{\n        \"type\": \"start\",\n        \"session_id\": \"6d8ba9ac-a515-49be-8bf4-cdef021a169d\",\n        \"user_id\": \"user01\",\n        \"text\": \"こんにちは\"\n    }'\n\n```\n\nSample response (streamed from the server):\n\n```sh\ndata: {\"type\": \"start\", \"session_id\": \"6d8ba9ac-a515-49be-8bf4-cdef021a169d\", \"user_id\": \"user01\", \"context_id\": \"c37ac363-5c65-4832-aa25-fd3bbbc1b1e7\", \"text\": null, \"voice_text\": null, \"avatar_control_request\": null, \"audio_data\": \"XXXX\", \"metadata\": {\"request_text\": \"こんにちは\"}}\n\ndata: {\"type\": \"chunk\", \"session_id\": \"6d8ba9ac-a515-49be-8bf4-cdef021a169d\", \"user_id\": \"user01\", \"context_id\": \"c37ac363-5c65-4832-aa25-fd3bbbc1b1e7\", \"text\": \"[face:joy]こんにちは！\", \"voice_text\": \"こんにちは！\", \"avatar_control_request\": {\"animation_name\": null, \"animation_duration\": null, \"face_name\": \"joy\", \"face_duration\": 4.0}, \"audio_data\": \"XXXX\", \"metadata\": {\"is_first_chunk\": true}}\n\ndata: {\"type\": \"chunk\", \"session_id\": \"6d8ba9ac-a515-49be-8bf4-cdef021a169d\", \"user_id\": \"user01\", \"context_id\": \"c37ac363-5c65-4832-aa25-fd3bbbc1b1e7\", \"text\": \"今日はどんなことをお手伝いしましょうか？\", \"voice_text\": \"今日はどんなことをお手伝いしましょうか？\", \"avatar_control_request\": {\"animation_name\": null, \"animation_duration\": null, \"face_name\": null, \"face_duration\": null}, \"audio_data\": \"XXXX\", \"metadata\": {\"is_first_chunk\": false}}\n\ndata: {\"type\": \"final\", \"session_id\": \"6d8ba9ac-a515-49be-8bf4-cdef021a169d\", \"user_id\": \"user01\", \"context_id\": \"c37ac363-5c65-4832-aa25-fd3bbbc1b1e7\", \"text\": \"[face:joy]こんにちは！今日はどんなことをお手伝いしましょうか？\", \"voice_text\": \"こんにちは！今日はどんなことをお手伝いしましょうか？\", \"avatar_control_request\": null, \"audio_data\": \"XXXX\", \"metadata\": {}}\n```\n\nTo continue the conversation, include the `context_id` provided in the `start` response in your next request.\n\n**NOTE:** When using the RESTful API, voice activity detection (VAD) must be performed client-side.\n\n**NOTE:** To protect API with API Key, set `api_key=API_KEY_YOU_MAKE` to AIAvatarHttpServer and send `Authorization: Bearer {API_KEY_YOU_MAKE}` as HTTP header from client.\n\n\n### 🔵 Dify-compatible API\n\n`AIAvatarHttpServer` provides a Dify-compatible `/chat-messages` endpoint (SSE streaming only).\nThis allows you to connect frontend applications that use Dify as their backend.\n\nFor more details, refer to the [Dify API Guide](https://docs.dify.ai/en/guides/application-publishing/developing-with-apis)\nor the API documentation of your self-hosted Dify application.\n\n\n### 🔌 WebSocket\n\nBelow is the simplest example of a server program:\n\n```python\nfrom fastapi import FastAPI\nfrom aiavatar.adapter.websocket.server import AIAvatarWebSocketServer\n\n# Create AIAvatar\naiavatar_app = AIAvatarWebSocketServer(\n    openai_api_key=OPENAI_API_KEY,\n    volume_db_threshold=-30,  # \u003c- Adjust for your audio env\n    debug=True\n)\n\n# Set router to FastAPI app\napp = FastAPI()\nrouter = aiavatar_app.get_websocket_router()\napp.include_router(router)\n```\n\nSave the above code as `server.py` and run it using:\n\n```sh\nuvicorn server:app\n```\n\n**NOTE:** When you specify `response_audio_chunk_size` in the `AIAvatarWebSocketServer` instance, the audio response will be streamed as PCM data chunks of the specified byte size. In this case, no WAVE header will be included in the response - you'll receive raw PCM audio data only.\n\n\nNext is the simplest example of a client program:\n\n```python\nimport asyncio\nfrom aiavatar.adapter.websocket.client import AIAvatarWebSocketClient\n\nclient = AIAvatarWebSocketClient()\nasyncio.run(client.start_listening(session_id=\"ws_session\", user_id=\"ws_user\"))\n```\n\nSave the above code as `client.py` and run it using:\n\n```sh\npython client.py\n```\n\nYou can now perform voice interactions just like when running locally.\n\n**NOTE:** When using the WebSocket API, voice activity detection (VAD) is performed on the server side, so clients can simply stream microphone input directly to the server.\n\n\n#### Connection and Disconnection Handling\n\nYou can register callbacks to handle WebSocket connection and disconnection events. This is useful for logging, session management, or custom initialization/cleanup logic.\n\n```python\n@aiavatar_app.on_connect\nasync def on_connect(request, session_data):\n    print(f\"Client connected: {session_data.id}\")\n    print(f\"User ID: {session_data.user_id}\")\n    print(f\"Session ID: {session_data.session_id}\")\n    \n    # Custom initialization logic\n    # e.g., load user preferences, initialize resources, etc.\n\n@aiavatar_app.on_disconnect\nasync def on_disconnect(session_data):\n    print(f\"Client disconnected: {session_data.id}\")\n    \n    # Custom cleanup logic\n    # e.g., save session data, release resources, etc.\n```\n\nThe `session_data` object contains information about the WebSocket session:\n\n- `id`: Unique session identifier\n- `user_id`: User identifier from the connection request\n- `session_id`: Session identifier from the connection request\n- Additional metadata passed during connection\n\n\n### 🟩 LINE Bot\n\nYou can build a LINE Bot using the LINE Messaging API.\n\n```python\n# NOTE: Register https://{your.domain}/webhook as the \"Webhook URL\" in LINE Developers Console\n\n# Create LINE Bot adapter\nfrom aiavatar.adapter.linebot.server import AIAvatarLineBotServer\naiavatar_app = AIAvatarLineBotServer(\n    openai_model=\"gpt-5.1\",\n    system_prompt=\"You are a cat.\",\n    openai_api_key=OPENAI_API_KEY,\n    channel_access_token=LINEBOT_CHANNEL_ACCESS_TOKEN,\n    channel_secret=LINEBOT_CHANNEL_SECRET,\n    image_download_url_base=\"https://{your.domain}\",\n    debug=True\n)\n\n# Create FastAPI app\nfrom fastapi import FastAPI\napp = FastAPI()\n\n# Set adapter endpoints\nrouter = aiavatar_app.get_api_router()\napp.include_router(router)\n```\n\n\nBy default, the LINE Messaging API user ID is used as the AIAvatarKit user ID. If you want to map it to your own AIAvatarKit user IDs, implement `edit_linebot_session` as shown below to update the session data.\n\n```python\nfrom aiavatar.adapter.linebot import LineBotSession\n@aiavatar_app.edit_linebot_session\nasync def edit_linebot_session(linebot_session: LineBotSession):\n    # Get user_id by LINE Bot user id\n    aiavatar_user_id = map_user_id(linebot_session.linebot_user_id)\n    # Set user_id to LINE Bot session\n    linebot_session.user_id = aiavatar_user_id\n```\n\nOther customization hooks are below.\n\n```python\n@aiavatar_app.preprocess_request\nasync def preprocess_request(request: STSRequest):\n    # Pre-process request before sending to LLM\n    # e.g. edit request text\n    request.text = \"Pre-processed: \" + request.text\n\n@aiavatar_app.preprocess_response\nasync def preprocess_response(response: STSResponse):\n    # Pre-process response before sending to LINE API\n    # e.g. edit response voice_text (not text)\n    response.voice_text = \"Pre-processed: \" + response.voice_text\n\n@aiavatar_app.process_avatar_control_request\nasync def process_avatar_control_request(avatar_control_request: AvatarControlRequest, reply_message_request: ReplyMessageRequest):\n    # Process facial expression\n    # e.g. set `sender` to the message in reply_message_request to change icon\n    face = avatar_control_request.face_name\n    if face:\n        reply_message_request.messages[0].sender = Sender(iconUrl=f\"https://your_domain/path/to/icon/{face}.png\")\n\n@aiavatar_app.on_send_error_message\nasync def on_send_error_message(reply_message_request: ReplyMessageRequest, linebot_session: LineBotSession, event: Event, ex: Exception):\n    # Pre-process error message\n    # e.g. edit error response\n    text = make_user_friendly_error_message(event, ex)\n    reply_message_request.messages[0] = TextMessage(text=text)\n\n@aiavatar_app.event(\"postback\")\nasync def handle_postback_event(event: Event, linebot_session: LineBotSession):\n    # Process event\n    # e.g. Register postback data\n    await register_data(linebot_session.user_id, event.postback.data)\n```\n\n\nSession data is stored in `aiavatar.db` via SQLite by default. To use PostgreSQL, create a `PostgreSQLLineBotSessionManager` and pass it to `AIAvatarLineBotServer` as `session_manager`.\n\n```python\n# Create PostgreSQLLineBotSessionManager\nfrom aiavatar.adapter.linebot.session_manager.postgres import PostgreSQLLineBotSessionManager\nlinebot_session_manager = PostgreSQLLineBotSessionManager(\n    host=DB_HOST,\n    port=DB_PORT,\n    dbname=DB_NAME,\n    user=DB_USER,\n    password=DB_PASSWORD\n)\n\naiavatar_app = AIAvatarLineBotServer(\n    openai_model=\"gpt-5.1\",\n    system_prompt=\"You are a cat.\",\n    openai_api_key=OPENAI_API_KEY,\n    channel_access_token=LINEBOT_CHANNEL_ACCESS_TOKEN,\n    channel_secret=LINEBOT_CHANNEL_SECRET,\n    image_download_url_base=\"https://{your.domain}\",\n    session_manager=linebot_session_manager,    # \u003c- Set PostgresSQL session manager\n    debug=True\n)\n```\n\n\n### STT / TTS Endpoints\n\nAIAvatarHttpServer provides REST API endpoints for Speech-to-Text (STT) and Text-to-Speech (TTS) functionality:\n\n#### STT Endpoint\n`POST /stt` - Converts audio to text.\n\n```python\nimport requests\n\n# Read audio file\nwith open(\"audio.wav\", \"rb\") as f:\n    audio_data = f.read()\n\n# Send to STT endpoint\nresponse = requests.post(\n    \"http://localhost:8000/stt\",\n    data=audio_data,\n    headers={\"Content-Type\": \"audio/wav\"}\n)\n\nprint(response.json())  # {\"text\": \"recognized speech\"}\n```\n\n#### TTS Endpoint\n`POST /tts` - Converts text to speech.\n\n```python\nimport requests\n\n# Send text to TTS endpoint\nresponse = requests.post(\n    \"http://localhost:8000/tts\",\n    json={\"text\": \"Hello, this is AI Avatar speaking\"}\n)\n\n# Save audio response\nwith open(\"output.wav\", \"wb\") as f:\n    f.write(response.content)\n```\n\n\n## 🛡️ Guardrail\n\nYou can apply guardrails to both requests and responses.\nGuardrails are custom implementations created by developers, and can block or replace an incoming request, or replace an outgoing response when certain conditions are met.\n\nBelow is the implementation method and how to apply guardrails.\n\n```python\nfrom aiavatar.sts.llm import Guardrail, GuardrailRespose\n\n# Define guardrails\nclass RequestGuardrail(Guardrail):\n    async def apply(self, context_id, user_id, text, files = None, system_prompt_params = None):\n        if text.lower() == \"problematic input\":\n            return GuardrailRespose(\n                guardrail_name=self.name,\n                is_triggered=True,\n                action=\"block\",\n                text=\"The problematic input has been blocked.\"  # Immediately returns this message to the user\n            )\n        elif text.lower() == \"hello\":\n            return GuardrailRespose(\n                guardrail_name=self.name,\n                is_triggered=True,\n                action=\"replace\",\n                text=\"こんにちは\"   # Replaces the original request text with this value\n            )\n        else:\n            return GuardrailRespose(\n                guardrail_name=self.name,\n                is_triggered=False\n            )\n\nclass ResponseGuardrail(Guardrail):\n    async def apply(self, context_id, user_id, text, files = None, system_prompt_params = None):\n        if \"ramen\" in text.lower():\n            return GuardrailRespose(\n                guardrail_name=self.name,\n                is_triggered=True,\n                action=\"replace\",\n                text=\"The problematic output has been blocked.\" # Emits an additional replacement chunk for the response\n            )\n        else:\n            return GuardrailRespose(\n                guardrail_name=self.name,\n                is_triggered=False\n            )\n\n# Apply guardrails\nservice.guardrails.append(RequestGuardrail(applies_to=\"request\"))\nservice.guardrails.append(ResponseGuardrail(applies_to=\"response\"))\n```\n\n**NOTE:** When multiple guardrails are defined, they run in parallel.\nProcessing stops when all guardrails have finished evaluating or when the first guardrail returns a response with `is_triggered=True`.\n\n**NOTE:** Response guardrails are evaluated only after the LLM response stream finishes.\nThis means the problematic output may be briefly visible to the user.\nWhen a response is received with `metadata.is_guardrail_triggered = true`, the client should handle this by replacing or modifying the output accordingly.\n\n\n## 🌎 Platform Guide\n\nAIAvatarKit is capable of operating on any platform that allows applications to hook into audio input and output. The platforms that have been tested include:\n\n- VRChat\n- cluster\n- Vket Cloud\n\nIn addition to running on PCs to operate AI avatars on these platforms, you can also create a communication robot by connecting speakers, a microphone, and, if possible, a display to a Raspberry Pi.\n\n### 🐈 VRChat\n\n* __2 Virtual audio devices (e.g. VB-CABLE) are required.__\n* __Multiple VRChat accounts are required to chat with your AIAvatar.__\n\n\n#### Get started\n\nFirst, run the commands below in python interpreter to check the audio devices.\n\n```sh\n$ python\n\n\u003e\u003e\u003e from aiavatar import AudioDevice\n\u003e\u003e\u003e AudioDevice().list_audio_devices()\n0: Headset Microphone (Oculus Virt\n    :\n6: CABLE-B Output (VB-Audio Cable\n7: Microsoft サウンド マッパー - Output\n8: SONY TV (NVIDIA High Definition\n    :\n13: CABLE-A Input (VB-Audio Cable A\n    :\n```\n\nIn this example,\n\n- To use `VB-Cable-A` for microphone for VRChat, index for `output_device` is `13` (CABLE-A Input).\n- To use `VB-Cable-B` for speaker for VRChat, index for `input_device` is `6` (CABLE-B Output). Don't forget to set `VB-Cable-B Input` as the default output device of Windows OS.\n\nThen edit `run.py` like below.\n\n```python\n# Create AIAvatar\naiavatar_app = AIAvatar(\n    openai_api_key=OPENAI_API_KEY,\n    input_device=6,     # Listen sound from VRChat\n    output_device=13,   # Speak to VRChat microphone\n)\n```\n\nRun it.\n\n```bash\n$ run.py\n```\n\nLaunch VRChat as desktop mode on the machine that runs `run.py` and log in with the account for AIAvatar. Then set `VB-Cable-A` to microphone in VRChat setting window.\n\nThat's all! Let's chat with the AIAvatar. Log in to VRChat on another machine (or Quest) and go to the world the AIAvatar is in.\n\n\n#### Face Expression\n\nAIAvatarKit controls the face expression by [Avatar OSC](https://docs.vrchat.com/docs/osc-avatar-parameters).\n\nLLM(ChatGPT/Claude/Gemini)\n↓ *response with face tag* `[face:joy]Hello!` or `\u003cface name=\"joy\" /\u003eHello!`\nAIAvatarKit(VRCFaceExpressionController)  \n↓ *osc* `FaceOSC=1`  \nVRChat(FX AnimatorController)  \n↓  \n😆\n\nSo at first, setup your avatar the following steps:\n\n1. Add avatar parameter `FaceOSC` (type: int, default value: 0, saved: false, synced: true).\n1. Add `FaceOSC` parameter to the FX animator controller.\n1. Add layer and put states and transitions for face expression to the FX animator controller.\n1. (option) If you use the avatar that is already used in VRChat, add input parameter configuration to avatar json.\n\n\nNext, use `VRChatFaceController`.\n\n```python\nfrom aiavatar.face.vrchat import VRChatFaceController\n\n# Setup VRChatFaceContorller\nvrc_face_controller = VRChatFaceController(\n    faces={\n        \"neutral\": 0,   # always set `neutral: 0`\n\n        # key = the name that LLM can understand the expression\n        # value = FaceOSC value that is set to the transition on the FX animator controller\n        \"joy\": 1,\n        \"angry\": 2,\n        \"sorrow\": 3,\n        \"fun\": 4\n    }\n)\n```\n\nLastly, add face expression section to the system prompt.\n\n```python\n# Make system prompt\nsystem_prompt = \"\"\"\n# Face Expression\n\n* You have following expressions:\n\n- joy\n- angry\n- sorrow\n- fun\n\n* If you want to express a particular emotion, please insert it at the beginning of the sentence like [face:joy].\n\nExample\n[face:joy]Hey, you can see the ocean! [face:fun]Let's go swimming.\n\"\"\"\n\n# Set them to AIAvatar\naiavatar_app = AIAvatar(\n    openai_api_key=OPENAI_API_KEY,\n    face_controller=vrc_face_controller,\n    system_prompt=system_prompt\n)\n```\n\nYou can test it not only through the voice conversation but also via the [REST API](#-restful-apis).\n\n\n### 🍓 Raspberry Pi\n\nNow writing... ✍️\n\n\n## ⚙️ Administration\n\nAIAvatarKit provides a built-in admin panel for monitoring, controlling, and evaluating your AI avatar from a web browser.\n\n### Admin Panel\n\nSet up the admin panel with a single function call. Once configured, access it at `/admin` on your server.\n\n```python\nfrom aiavatar.admin import setup_admin_panel\n\nsetup_admin_panel(\n    app,\n    adapter=aiavatar_app,\n    evaluator=evaluator,                    # Optional: If omitted, the pipeline LLM settings are used\n    character_service=character_service,    # Optional: If using CharacterService\n    character_id=YOUR_CHARACTER_ID,         # Optional: Required if character_service is set\n    api_key=\"your-api-key\"                  # Optional: If omitted, no authentication is required\n)\n```\n\nThe admin panel includes:\n\n- **Metrics** — Real-time performance metrics for the STS pipeline\n- **Logs** — Conversation logs with voice playback\n- **Control** — Send speech and conversation messages to the avatar\n- **Config** — Adjust pipeline, VAD, STT, LLM, TTS, and adapter settings at runtime\n- **Evaluation** — Run dialog evaluation scenarios\n- **Character** — Manage character info, weekly schedule, daily schedules, diaries, and users (requires `character_service`)\n\nTo protect the admin panel with Basic authentication:\n\n```python\nsetup_admin_panel(\n    app,\n    adapter=aiavatar_app,\n    api_key=\"your-api-key\",\n    basic_auth_username=\"admin\",\n    basic_auth_password=\"your-password\",\n)\n```\n\nYou can also supply your own HTML to fully customize the admin page:\n\n```python\ncustom_html = open(\"my_admin.html\").read()\n\nsetup_admin_panel(\n    app,\n    adapter=aiavatar_app,\n    html=custom_html,       # Use your own HTML instead of the built-in template\n)\n```\n\n### REST API\n\nAll admin panel features are also available as REST API endpoints. See the interactive API documentation at `/docs` on your server for full details on request/response schemas.\n\n### 📈 Observability\n\nYou can monitor the entire sequence - what requests are sent to the LLM, how they are interpreted, which tools are invoked, and what responses are generated from specific results or data - to support AIAvatar quality improvements and governance.\n\nSince AIAvatarKit lets you replace the OpenAI client module with an alternative, you can leverage that capability to integrate with [Langfuse](https://langfuse.com).\n\n```sh\npip install langfuse\n```\n\n```sh\nexport LANGFUSE_SECRET_KEY=sk-lf-XXXXXXXX\nexport LANGFUSE_PUBLIC_KEY=pk-lf-XXXXXXXX\nexport LANGFUSE_HOST=http://localhost:3000\n```\n\n```python\nfrom langfuse.openai import openai as langfuse_openai\nllm = ChatGPTService(\n    openai_api_key=OPENAI_API_KEY,\n    system_prompt=\"You are a helpful assistant.\",\n    model=\"gpt-4.1\",\n    custom_openai_module=langfuse_openai,   # Set langfuse OpenAI compatible client module\n)\n```\n\n\n## 🦜 AI Agent\n\nAIAvatarKit is not just a framework for creating chatty AI characters — it is designed to support agentic characters that can interact with APIs and external data sources (RAG).\n\n### ⚡️ Tool Call\n\nRegister tool with spec by `@aiavatar_app.sts.llm.tool`. The spec should be in the format for each LLM.\n\n```python\n# Spec (for ChatGPT)\nweather_tool_spec = {\n    \"type\": \"function\",\n    \"function\": {\n        \"name\": \"get_weather\",\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"location\": {\"type\": \"string\"}\n            },\n        },\n    }\n}\n\n# Implement tool and register it with spec\n@aiavatar_app.sts.llm.tool(weather_tool_spec)\nasync def get_weather(location: str = None):\n    weather = await weather_api(location=location)  # Call weather API\n    return weather  # {\"weather\": \"clear\", \"temperature\": 23.4}\n```\n\nAlternatively, register the same tool programmatically:\n\n```python\naiavatar_app.sts.llm.add_tool(\n    Tool(\"get_weather\", weather_tool_spec, get_weather)\n)\n```\n\n**Note**: When you register a tool with `add_tool`, the spec is automatically converted to the correct format for GPT, Gemini, or Claude, so you can define it once and use it everywhere.\n\n\nBefore creating your own tools, start with the example tools:\n\n```python\n# Google Search\nfrom examples.tools.gemini_websearch import GeminiWebSearchTool\naiavatar_app.sts.llm.add_tool(GeminiWebSearchTool(gemini_api_key=GEMINI_API_KEY))\n\n# Web Scraper\nfrom examples.tools.webscraper import WebScraperTool\naiavatar_app.sts.llm.add_tool(WebScraperTool())\n```\n\n\n### ⌛️ Tool Call with Streaming Progress\n\nSometimes you may want to provide feedback to the user when a tool takes time to execute. AIAvatarKit supports tools that return stream responses (via `AsyncGenerator`), which allows you to integrate advanced and costly operations — such as interactions with AI Agent frameworks — into real-time voice conversations without compromising the user experience.\n\nHere’s an example implementation. Intermediate progress is yielded with the second return value set to `False`, and the final result is yielded with `True`.\n\n```python\n@service.tool(weather_tool_spec)\nasync def get_weather_stream(location: str):\n    # Progress: Geocoding\n    yield {\"message\": \"Resolving location\"}, False\n    geocode = await geocode_api(location=location)\n\n    # Progress: Weather\n    yield {\"message\": \"Calling weather api\"}, False\n    weather = await weather_api(geocode=geocode)  # Call weather API\n\n    # Final result (yield with `True`)\n    yield {\"weather\": \"clear\", \"temperature\": 23.4}, True\n```\n\nOn the user side, the first value in each yield will be streamed as a `progress` response under the `ToolCall` response type.\n\nAdditionally, you can yield string values directly to provide immediate voice feedback to the user during processing:\n\n```python\n@service.tool(weather_tool_spec)\nasync def get_weather_stream(location: str):\n    # Provide voice feedback during processing\n    yield \"Converting locaton to geo code. Please wait a moment.\"\n    geocode = await geocode_api(location=location)\n    \n    yield \"Getting weather information.\"\n    weather = await weather_api(geocode=geocode)\n    \n    # Final result\n    yield {\"weather\": \"clear\", \"temperature\": 23.4}, True\n```\n\nWhen you yield a string (str) value, the AI avatar will speak that text while continuing to process the request.\n\n\n### 🔄 Background Tool Execution\n\nFor tools that take a long time to complete (e.g., AI agent calls, complex API orchestrations), AIAvatarKit supports **background execution**. Instead of blocking the conversation, the avatar immediately acknowledges the request and notifies the user when the result is ready via a callback.\n\nTo enable background execution, register an `on_completed` callback on the tool. This is the only requirement — the base `Tool` class handles task management, `task_id` generation, and metadata tracking automatically.\n\n```python\nfrom aiavatar.sts.llm import Tool\n\n# Define tool as usual\nheavy_task_spec = {\n    \"type\": \"function\",\n    \"function\": {\n        \"name\": \"run_heavy_task\",\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"query\": {\"type\": \"string\"}\n            },\n            \"required\": [\"query\"]\n        },\n    }\n}\n\nasync def run_heavy_task(query: str, metadata: dict = None):\n    result = await some_slow_api(query)  # Takes a long time\n    return {\"answer\": result}\n\ntool = Tool(\"run_heavy_task\", heavy_task_spec, run_heavy_task)\n\n# Enable background execution by registering on_completed callback\n@tool.on_completed\nasync def on_completed(result, metadata):\n    # result: return value from the tool function (or None on error)\n    # metadata: dict containing task_id, user_id, context_id, submitted_at, arguments, etc.\n    answer = result[\"answer\"]\n    user_id = metadata[\"user_id\"]\n    context_id = metadata[\"context_id\"]\n    session_id = aiavatar_app.get_session_by_user_id(user_id).id\n\n    async for resp in aiavatar_app.sts.invoke(\n        STSRequest(\n            session_id=session_id,\n            user_id=user_id,\n            context_id=context_id,\n            text=f\"Here is the result of the task:\\n\\n{answer}\",\n            wait_in_queue=True,\n            skip_quick_response=True,\n        )\n    ):\n        await aiavatar_app.handle_response(resp)\n\nllm.add_tool(tool)\n```\n\nWhen background execution is enabled:\n\n1. The tool function is called and runs in the background as an `asyncio.Task`\n2. The avatar immediately responds with `immediate_message` (customizable) and a `task_id`\n3. When the function completes, `on_completed` is called with the result and metadata\n\nYou can customize the immediate message:\n\n```python\ntool = Tool(\n    \"run_heavy_task\", heavy_task_spec, run_heavy_task,\n    immediate_message=\"Got it! I'll work on that and let you know when it's done.\"\n)\n```\n\nOptionally, register an `on_submitted` callback to be notified when the task is accepted:\n\n```python\n@tool.on_submitted\nasync def on_submitted(task_id, metadata):\n    print(f\"Task {task_id} submitted\")\n```\n\n#### Background Timeout (Hybrid Mode)\n\nSometimes a tool *might* complete quickly but *could* take a long time. With `background_timeout`, AIAvatarKit tries synchronous execution first and falls back to background execution only if the timeout is exceeded.\n\n```python\ntool = Tool(\n    \"run_task\", task_spec, run_task,\n    background_timeout=3.0  # Try sync for 3 seconds, then go background\n)\n\n@tool.on_completed\nasync def on_completed(result, metadata):\n    # Called only when the task didn't complete within the timeout\n    print(f\"Background result: {result}\")\n```\n\n- If the tool completes within `background_timeout` seconds → result is returned directly (same as synchronous mode)\n- If the tool exceeds the timeout → switches to background mode, returns `immediate_message`, and calls `on_completed` when done\n\n**Note**: `on_completed` (background execution) and `AsyncGenerator` (streaming progress) are mutually exclusive. A tool should use one pattern or the other.\n\n\n### 📋 Tool Response Formatter (Direct Response)\n\nBy default, after a tool executes, the result is passed back to the LLM to generate a human-friendly response (2nd LLM call). However, in some cases you may want to **bypass the LLM and speak the tool result directly**:\n\n- **Accuracy**: Critical information (e.g., order details, reservation IDs) that must not be paraphrased or hallucinated\n- **Latency**: Eliminating the 2nd LLM call for faster response times\n\nUse the `@response_formatter` decorator to define a function that converts the tool result into the exact text to speak. When a `response_formatter` is set, the 2nd LLM call is skipped entirely, and the formatted text is spoken directly.\n\n```python\n@llm.tool(weather_tool_spec)\nasync def get_weather(location: str = None):\n    weather = await weather_api(location=location)\n    return weather  # {\"weather\": \"clear\", \"temperature\": 23.4}\n\n# Register response_formatter to speak the result directly\n@llm.tools[\"get_weather\"].response_formatter\ndef format_weather(result, arguments):\n    return f\"The weather in {arguments['location']} is {result['weather']}, with a temperature of {result['temperature']} degrees.\"\n```\n\nThe formatter receives two arguments:\n\n| Argument | Description |\n|----------|-------------|\n| `result` | The dict returned by the tool function |\n| `arguments` | The dict of arguments passed to the tool by the LLM |\n\nThe tool call and its result are still saved to conversation context, so follow-up questions like \"What was the temperature again?\" work naturally. The formatted text is stored as the assistant's response.\n\n**Note**: Tools without a `response_formatter` continue to work as before (2nd LLM call generates the response). You can mix both patterns — some tools with formatters and others without.\n\n\n### 🪄 Dynamic Tool Call\n\nAIAvatarKit supports **dynamic Tool Calls**.\nWhen many tools are loaded up-front, it becomes harder to make the model behave as intended and your system instructions explode in size. With AIAvatarKit’s **Dynamic Tool Call** mechanism you load **only the tools that are actually needed at the moment**, eliminating that complexity.\n\nThe overall flow is illustrated below.\n\n![Dynamic Tool Call Mechanism](documents/images/dynamic_tool_call.png)\n\n#### 1. Create the tool definitions and implementations  \n*(exactly the same as with ordinary tools)*\n\n```python\n# Weather\nget_weather_spec = {\n    \"type\": \"function\",\n    \"function\": {\n        \"name\": \"get_weather\",\n        \"description\": \"Get weather info at the specified location\",\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"location\": {\"type\": \"string\"}\n            }\n        },\n    }\n}\n\nasync def get_weather(location: str):\n    resp = await weather_api(location)\n    return resp.json() # e.g. {\"weather\": \"clear\", \"temperature\": 23.4}\n\n# Web Search\nsearch_web_spec = {\n    \"type\": \"function\",\n    \"function\": {\n        \"name\": \"search_web\",\n        \"description\": \"Search info from the internet websites\",\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"query\": {\"type\": \"string\"}\n            }\n        },\n    }\n}\nasync def search_web(query: str) -\u003e str:\n    resp = await web_search_api(query)\n    return resp.json() # e.g. {\"results\": [{...}]}\n```\n\n#### 2. Register the tools as dynamic in the AIAvatarKit LLM service\n\nSetting `is_dynamic=True` tells the framework not to expose the tool by default;\nAIAvatarKit will inject it only when the Trigger Detection Tool decides the tool is relevant.\nYou can also supply an `instruction` string that will be spliced into the system prompt on-the-fly.\n\n```python\nfrom aiavatar.sts.llm import Tool\n\nllm = aiavatar_app.sts.llm\n\n# Turn on Dynamic Tool Mode\nllm.use_dynamic_tools = True\n\n# Register as Dynamic Tools\nllm.tools[\"get_weather\"] = Tool(\n    \"get_weather\",\n    get_weather_spec,\n    get_weather,\n    instruction=(\n        \"## Use of `get_weather`\\n\\n\"\n        \"Call this tool to obtain current weather or a forecast. \"\n        \"Argument:\\n\"\n        \"- `location`: city name or geo-hash.\"\n    ),\n    is_dynamic=True,\n)\n\nllm.tools[\"search_web\"] = Tool(\n    \"search_web\",\n    search_web_spec,\n    search_web,\n    instruction=(\n        \"## Use of `search_web`\\n\\n\"\n        \"Call this tool to look up information on the public internet. \"\n        \"Argument:\\n\"\n        \"- `query`: keywords describing what you want to find.\"\n    ),\n    is_dynamic=True,\n)\n```\n\nOr, register via `add_tool`.\n\n```python\n# Difine tool without `is_dynamic` for other use cases\nweather_tool = Tool(\"get_weather\", get_weather_spec, get_weather, instruction=\"...\")\n\n# Register tool via `add_tool` with `is_dynamic`\nllm.add_tool(weather_tool, is_dynamic=True)\n```\n\n\n#### 3. Tweak the system prompt so the model knows how to use tools\n\nAppend a concise “How to use external tools” section (example below).\nReplace the example tools with those your application actually relies on for smoother behaviour.\n\n\n```md\n## Use of External Tools\n\nWhen external tools, knowledge, or data are required to process a user's request, use the appropriate tools.  \nThe following rules **must be strictly followed** when using tools.\n\n### Arguments\n\n- Use only arguments that are **explicitly specified by the user** or that can be **reliably inferred from the conversation history**.\n- **If information is missing**, ask the user for clarification or use other tools to retrieve the necessary data.\n- **It is strictly forbidden** to use values as arguments that are not based on the conversation.\n\n### Tool Selection\n\nWhen a specialized tool is available for a specific purpose, use that tool.  \nIf you can use only `execute_external_tool`, use it.\n\nExamples where external tools are needed:\n\n- Retrieving weather information  \n- Retrieving memory from past conversations  \n- Searching for, playing, or otherwise controlling music  \n- Performing web searches  \n- Accessing real-world systems or data to provide better solutions\n```\n\nWith these three steps, your AI agent stays lean—loading only what it needs—while still having immediate access to a rich arsenal of capabilities whenever they’re truly required.\n\n\n#### Custom Tool Repository\n\nBy default AIAvatarKit simply hands the **entire list of dynamic tools** to the LLM and lets the model decide which ones match the current context. This approach works for a moderate number of tools, but the size of the prompt places a hard limit on how many candidates you can include.\n\nFor larger-scale systems, pair AIAvatarKit with a retrieval layer (e.g., a vector-search index) so that, out of thousands of available tools, only the handful that are truly relevant are executed.\n\nAIAvatarKit supports this pattern through the `get_dynamic_tools` hook.\nRegister an async function decorated with `@llm.get_dynamic_tools`; it should return a list of **tool specification objects** for the current turn.\n\n```python\n@llm.get_dynamic_tools\nasync def my_get_dynamic_tools(messages: list, metadata: dict) -\u003e list:\n    # Retrieve candidate tools from your vector database (or any other store)\n    tools = await search_tools_from_vector_db(messages, metadata)\n    # Extract and return the spec objects (not the implementations)\n    return [t.spec for t in tools]\n```\n\n### 🔌 MCP\n\nAIAvatarKit supports tools provided as MCP.\n\nFirst, install the required `FastMCP` dependency.\n\n```sh\npip install fastmcp\n```\n\nThe following steps show how to retrieve tools from MCP servers and register them to `LLMService`.\n\nBoth Streamable HTTP and standard I/O are supported. The simplest approach is shown in `mcp1` and `mcp3`, but you can also add authentication headers as in `mcp2`, filter tools to only what you need, or customize parts of the schema or execution logic.\n\n```python\nfrom aiavatar.sts.llm.chatgpt import ChatGPTService\nllm = ChatGPTService(openai_api_key=OPENAI_API_KEY)\n\nfrom aiavatar.sts.llm.tools.mcp import StreamableHttpMCP, StdioMCP\n\n# MCP Server\nmcp1 = StreamableHttpMCP(url=MCP1_URL)\nmcp1.for_each_tool = llm.add_tool\n\n# MCP Server with Auth\nmcp2 = StreamableHttpMCP(url=MCP2_URL, headers={\"Authorization\": f\"Bearer {MCP_JWT}\"})\n@mcp2.for_each_tool\ndef mcp2_tools(tool: Tool):\n    # Do something here (e.g. edit schema or func)\n    llm.add_tool(tool)\n\n# MCP Server (Std I/O)\nmcp3 = StdioMCP(server_script=\"weather.py\") # supports .py and .js\nmcp3.for_each_tool = llm.add_tool\n```\n\n### 🛠️ Built-in Tools\n\nYou can use the following tools out of the box 📦.\n\n- 🔍 Web Search\n    - Gemini Search\n    - OpenAI Search\n    - Grok Search\n- 🌏 Web Scraper\n- 🖼️ Image Generation\n    - 🍌 Nano Banana\n    - 🐓 Selfie\n\n```python\n# Web Search\nfrom aiavatar.sts.llm.tools.gemini_websearch import GeminiWebSearchTool\ngoogle_search_tool = GeminiWebSearchTool(gemini_api_key=GEMINI_API_KEY)\nllm.add_tool(google_search_tool)\n\nfrom aiavatar.sts.llm.tools.openai_websearch import OpenAIWebSearchTool\nweb_search_tool = OpenAIWebSearchTool(openai_api_key=OPENAI_API_KEY)\nllm.add_tool(web_search_tool)\n\nfrom aiavatar.sts.llm.tools.grok_search import GrokSearchTool\ngrok_web_search_tool = GrokSearchTool(xai_api_key=XAI_API_KEY)\nllm.add_tool(grok_web_search_tool)\n\n# Web Scraper\nfrom aiavatar.sts.llm.tools.webscraper import WebScraperTool\nwebscraper_tool = WebScraperTool()\n# webscraper_tool = WebScraperTool(openai_api_key=OPENAI_API_KEY, return_summary=True)  # Provides summary instead of full innerText (recommended)\nllm.add_tool(webscraper_tool)\n\n# Image Generation\nfrom aiavatar.sts.llm.tools.nanobanana import NanoBananaTool\nnanobanana_tool = NanoBananaTool(gemini_api_key=GEMINI_API_KEY)\nllm.add_tool(nanobanana_tool)\n\nfrom aiavatar.sts.llm.tools.nanobanana import NanoBananaSelfieTool\nselfie_tool = NanoBananaSelfieTool(gemini_api_key=GEMINI_API_KEY, reference_image=image_bytes_or_image_url_of_file_api)\nllm.add_tool(selfie_tool)\n```\n\n\n### 🦞 OpenClaw\n\n`OpenClawTool` integrates [OpenClaw](https://openclaw.ai), a versatile AI agent, as a tool for your avatar. When the LLM determines that the user's request requires autonomous task execution (web search, data analysis, code execution, etc.), it delegates the task to OpenClaw.\n\n```python\nfrom aiavatar.sts.llm.tools.openclaw_tool import OpenClawTool\nopenclaw_tool = OpenClawTool(\n    openclaw_api_key=OPENCLAW_API_KEY,\n    openclaw_base_url=OPENCLAW_BASE_URL,\n    openclaw_session_key=\"agent:main:main\",\n    debug=True\n)\nllm.add_tool(openclaw_tool)\n```\n\nWhen `on_completed` is registered, OpenClaw runs asynchronously in the background — the avatar immediately acknowledges the request and notifies the user when the result is ready. The approach for delivering the result depends on your adapter.\n\n#### Push-based delivery (WebSocket / Local)\n\nFor adapters that support server-initiated messages, use `on_completed` to push the result back through the pipeline:\n\n```python\n@openclaw_tool.on_completed\nasync def on_completed(result, metadata):\n    answer = result[\"answer\"]\n    user_id = metadata[\"user_id\"]\n    context_id = metadata[\"context_id\"]\n    session_id = aiavatar_app.get_session_by_user_id(user_id).id\n\n    async for resp in aiavatar_app.sts.invoke(\n        STSRequest(\n            session_id=session_id,\n            user_id=user_id,\n            context_id=context_id,\n            text=f\"$OpenClaw has returned a response. Please relay the following to the user:\\n\\n{answer}\",\n            wait_in_queue=True,\n            skip_quick_response=True,\n        )\n    ):\n        await aiavatar_app.handle_response(resp)\n```\n\n#### Polling-based delivery (HTTP)\n\nFor HTTP adapters where the SSE stream has already closed by the time the background task completes, store results in a buffer and let the client poll for them. The tool returns a `task_id` in its response for this purpose.\n\nRegister callbacks to track task lifecycle:\n\n```python\nimport time as time_module\ntask_results = {}\nTASK_TIMEOUT = 300  # 5 minutes\n\n@openclaw_tool.on_submitted\nasync def on_submitted(task_id: str, metadata: dict):\n    task_results[task_id] = {\n        \"task_id\": task_id,\n        \"submitted_at\": metadata.get(\"submitted_at\", time_module.time()),\n        \"answer\": None,\n    }\n\n@openclaw_tool.on_completed\nasync def on_completed(result, metadata):\n    task_id = metadata[\"task_id\"]\n    task_results[task_id][\"answer\"] = result[\"answer\"]\n```\n\nAdd a polling endpoint for the client to retrieve results:\n\n```python\n@app.get(\"/tasks/{task_id}\")\nasync def get_task_result(task_id: str):\n    result = task_results.get(task_id)\n    if result is None:\n        return Response(status_code=204)\n    if result[\"answer\"]:\n        task_results.pop(task_id, None)\n        return {\"task_id\": task_id, \"answer\": result[\"answer\"], \"status\": \"completed\"}\n    if time_module.time() - result[\"submitted_at\"] \u003e TASK_TIMEOUT:\n        task_results.pop(task_id, None)\n        return {\"task_id\": task_id, \"answer\": None, \"status\": \"timeout\"}\n    return Response(status_code=204)\n```\n\nThe client receives the `task_id` from the avatar's immediate response and polls `GET /tasks/{task_id}` until it gets a result (`status: \"completed\"`) or a timeout (`status: \"timeout\"`). A `204` response means the task is still in progress.\n\nOnce the client retrieves the answer, it can send it back to the avatar as a new request, for example `f\"$OpenClaw has returned a response. Please relay the following to the user:\\n\\n{answer}\"`, to have the avatar speak the result aloud.\n\n\n## 🧪 Evaluation\n\nAIAvatarKit includes a comprehensive evaluation framework for testing and assessing AI avatar conversations. The `DialogEvaluator` enables scenario-based conversation execution with automatic evaluation capabilities.\n\n### Features\n\n- **Scenario Execution**: Run predefined dialog scenarios against your AI system\n- **Turn-by-Turn Evaluation**: Evaluate each conversation turn against specific criteria\n- **Goal Assessment**: Evaluate overall scenario objective achievement\n- **Result Management**: Save, load, and display evaluation results\n\n### Basic Usage\n\n```python\nimport asyncio\nfrom aiavatar.eval.dialog import DialogEvaluator, Scenario, Turn\nfrom aiavatar.sts.llm.chatgpt import ChatGPTService\n\nasync def main():\n    # Initialize LLM services\n    llm = ChatGPTService(api_key=\"your_api_key\")\n    evaluation_llm = ChatGPTService(api_key=\"your_api_key\")\n    \n    # Create evaluator\n    evaluator = DialogEvaluator(\n        llm=llm,                    # LLM for conversation\n        evaluation_llm=evaluation_llm  # LLM for evaluation\n    )\n    \n    # Define scenario\n    scenario = Scenario(\n        name=\"Order tracking support\",\n        goal=\"Provide efficient and helpful customer service for order tracking inquiries\",\n        turns=[\n            Turn(\n                input_text=\"Hello, I need help with my order\",\n                evaluation_criteria=\"Responds politely and shows willingness to help\"\n            ),\n            Turn(\n                input_text=\"My order number is 12345\",\n                evaluation_criteria=\"Acknowledges the order number and proceeds appropriately\"\n            )\n        ]\n    )\n    \n    # Run evaluation\n    results = await evaluator.run(\n        dataset=[scenario],\n        detailed=True,                # Enable turn-by-turn evaluation\n        overwrite_execution=False,    # Skip if already executed\n        overwrite_evaluation=False    # Skip if already evaluated\n    )\n    \n    # Display results\n    evaluator.print_results(results)\n    \n    # Save results\n    evaluator.save_results(results, \"evaluation_results.json\")\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\nExample Output:\n\n```\n=== Scenario 1 ===\nGoal: Provide helpful customer support\n\nTurn 1:\n  Input: Hello, I need help with my order\n  Actual Output: Hello! I'd be happy to help you with your order. Could you please provide your order number?\n  Result: ✓ PASS\n  Reason: The response is polite, helpful, and appropriately asks for the order number.\n\nTurn 2:\n  Input: My order number is 12345\n  Actual Output: Thank you for providing order number 12345. Let me look that up for you.\n  Result: ✓ PASS\n  Reason: Acknowledges the order number and shows willingness to help.\n\nSummary: 2/2 turns passed (100.0%)\n\n=== Overall Scenario Evaluation ===\nGoal Achievement: ✓ SUCCESS\nReason: The AI successfully provided helpful customer support by responding politely and efficiently handling the order inquiry.\n```\n\n### File-Based Evaluation\n\nLoad scenarios from JSON files:\n\n```json\n{\n  \"scenarios\": [\n    {\n      \"goal\": \"Basic greeting and assistance\",\n      \"turns\": [\n        {\n          \"input_text\": \"Hello\",\n          \"expected_output\": \"Friendly greeting\",\n          \"evaluation_criteria\": \"Responds warmly and appropriately\"\n        }\n      ]\n    }\n  ]\n}\n```\n\n```python\n# Load and evaluate from file\nresults = await evaluator.run(dataset=\"test_scenarios.json\")\n\n# Save results back to file\nevaluator.save_results(results, \"results.json\")\n```\n\n### Configuration Options\n\n```python\n# Execution modes\nresults = await evaluator.run(\n    dataset=scenarios,\n    detailed=True,                # Turn-by-turn evaluation\n    overwrite_execution=True,     # Re-run conversations\n    overwrite_evaluation=True     # Re-evaluate results\n)\n\n# Simple mode (scenario-level evaluation only)\nresults = await evaluator.run(\n    dataset=scenarios,\n    detailed=False\n)\n```\n\n### Use via Config API\n\nYou can evaluate scenario on the fly via Config API:\n\n```python\n# Make evaluator\nfrom aiavatar.eval.dialog import DialogEvaluator\neval_llm = ChatGPTService(openai_api_key=OPENAI_API_KEY)\nevaluator = DialogEvaluator(llm=aiavatar_app.sts.llm, evaluation_llm=eval_llm)\n\n# Activate Config API\nfrom aiavatar.admin.config import ConfigAPI\nconfig_router = ConfigAPI(aiavatar_app.sts, evaluator=evaluator).get_router()   # Set evaluator here\napp.include_router(config_router)\n```\n\n### Logic-based evaluation\n\nIn addition to LLM-based evaluation using `evaluation_criteria`, you can evaluate more explicitly using custom logic functions.\n\n```python\n# Make evaluation function(s)\ndef evaluate_weather_tool_call(output_text, tool_call, evaluation_criteria, result, eval_result_text):\n    if tool_call is not None and tool_call.name != \"get_weather\":\n        # Overwrite result and reason\n        return False, f\"Incorrect tool call: {tool_call.name}\"\n    else:\n        # Pass through\n        return result, eval_result_text\n\n# Register evaluation function(s)\nevaluator = DialogEvaluator(\n    llm=aiavatar_app.sts.llm,\n    evaluation_llm=eval_llm,\n    evaluation_functions={\"evaluate_weather_tool_call_func\": evaluate_weather_tool_call}\n)\n\n# Use evaluation function in scenario\nscenario = Scenario(\n    turns=[\n        Turn(input_text=\"Hello\", expected_output_text=\"Hi\", evaluation_criteria=\"Greeting\"),\n        Turn(input_text=\"What is the weather in Tokyo?\", expected_output_text=\"It's sunny.\", evaluation_criteria=\"Answer the weather based on the result of calling get_weather tool.\", evaluation_function_name=\"evaluate_weather_tool_call_func\"),\n    ],\n    goal=\"Answer the weather in Tokyo based on the result of get_weather.\"\n)\n```\n\n\n## 🤿 Deep dive\n\nAdvanced usases.\n\n\n### 🐘 PostgreSQL\n\nYou can use PostgreSQL instead of the default SQLite. We strongly recommend using PostgreSQL in production environments for its scalability and performance benefits from asynchronous processing.\n\nTo use PostgreSQL, install asyncpg and create a `PostgreSQLPoolProvider` to manage the shared connection pool. Then pass it to the constructors of the components that need database access.\n\n\n```sh\npip install asyncpg\n```\n\n```python\n# DB_CONNECTION_STR = \"postgresql://{user}:{password}@{host}:{port}/{databasename}\"\nDB_CONNECTION_STR = \"postgresql://postgres:postgres@127.0.0.1:5432/aiavatar\"\n\n# PoolProvider\nfrom aiavatar.database.postgres import PostgreSQLPoolProvider\npool_provider = PostgreSQLPoolProvider(\n    connection_str=DB_CONNECTION_STR,\n    # max_size=20,  # Max connection count (default: 20)\n    # min_size=5    # Min connection count (default: 5)\n)\n\n# Character\nfrom aiavatar.character import CharacterService\ncharacter_service = CharacterService(\n    openai_api_key=OPENAI_API_KEY,\n    db_pool_provider=pool_provider,     # Creates PostgreSQLCharacterRepository and PostgreSQLActivityRepository internally\n)\n\n# LLM\nfrom aiavatar.sts.llm.context_manager.postgres import PostgreSQLContextManager\nllm = ChatGPTService(\n    openai_api_key=OPENAI_API_KEY,\n    system_prompt=SYSTEM_PROMPT,\n    context_manager=PostgreSQLContextManager(\n        get_pool=pool_provider.get_pool # Set `get_pool` to PostgreSQLContextManager","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuezo%2Faiavatarkit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fuezo%2Faiavatarkit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fuezo%2Faiavatarkit/lists"}