{"id":24061041,"url":"https://github.com/akikurisu/unichat","last_synced_at":"2025-04-23T07:53:50.546Z","repository":{"id":231492160,"uuid":"781889307","full_name":"AkiKurisu/UniChat","owner":"AkiKurisu","description":"一个用于在Unity中开发在线和离线聊天机器人的管线。A pipeline designed to create online and offline chat-bot in Unity. ","archived":false,"fork":false,"pushed_at":"2024-04-12T16:28:15.000Z","size":1356,"stargazers_count":15,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-04-12T23:59:32.783Z","etag":null,"topics":["agent","chat","chatbot","conversational-ai","conversational-bots","unity"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AkiKurisu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-04-04T08:31:35.000Z","updated_at":"2024-04-15T13:37:25.757Z","dependencies_parsed_at":"2024-04-12T17:35:00.080Z","dependency_job_id":null,"html_url":"https://github.com/AkiKurisu/UniChat","commit_stats":null,"previous_names":["akikurisu/unichat"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AkiKurisu%2FUniChat","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AkiKurisu%2FUniChat/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AkiKurisu%2FUniChat/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AkiKurisu%2FUniChat/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AkiKurisu","download_url":"https://codeload.github.com/AkiKurisu/UniChat/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250395187,"owners_count":21423376,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent","chat","chatbot","conversational-ai","conversational-bots","unity"],"created_at":"2025-01-09T07:16:25.782Z","updated_at":"2025-04-23T07:53:50.537Z","avatar_url":"https://github.com/AkiKurisu.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# UniChat\n\nA pipeline for creating online and offline chat-bot in Unity.\n\n\u003cimg src=\"./Docs/Images/Icon.png\" width=\"256\"/\u003e\n\n\u003c/div\u003e\n\n  - [Introduction](#introduction)\n  - [Install](#install)\n  - [Core pipeline](#core-pipeline)\n    - [Quick use](#quick-use)\n    - [Embedding model](#embedding-model)\n  - [Chain](#chain)\n    - [Combined with the core pipeline](#combined-with-the-core-pipeline)\n    - [Stack Trace](#stack-trace)\n  - [Middleware](#middleware)\n    - [Text to Speech](#text-to-speech)\n    - [Speech to Text](#speech-to-text)\n    - [Sub-classifier](#sub-classifier)\n    - [Tool Use](#tool-use)\n  - [Demo](#demo)\n    - [Download](#download)\n    - [Features](#features)\n      - [Personalization: Character Cards](#personalization-character-cards)\n  - [More Showcase](#more-showcase)\n  - [Quote](#quote)\n    - [Blogs](#blogs)\n    - [Repo](#repo)\n    - [Paper](#paper)\n\n\n\n\n## Introduction\n\nWith the release of `Unity.Sentis`, we can use some neural network models at Runtime, including the text embedding model for natural language processing.\n\nAlthough chatting with AI is nothing new, in games, how to design a conversation that does not deviate from the developer's ideas but is more flexible is a difficult point.\n\n`UniChat` is based on `Unity.Sentis` and text vector embedding technology, which enables \u003cb\u003eoffline mode\u003c/b\u003e to search text content based on vector databases.\n\nOf course, if you use the online mode, `UniChat` also includes a chain toolkit based on [LangChain](https://github.com/langchain-ai/langchain) to quickly embed LLM and Agent in the game.\n\n## Install\n\n1. Add the following dependencies in `manifest.json`:\n```json\n{\n  \"dependencies\": {\n    \"com.unity.addressables\": \"1.21.20\",\n    \"com.unity.burst\": \"1.8.13\",\n    \"com.unity.collections\": \"2.2.1\",\n    \"com.unity.nuget.newtonsoft-json\": \"3.2.1\",\n    \"com.unity.sentis\": \"1.3.0-pre.3\",\n    \"com.cysharp.unitask\": \"2.5.3\"\n    }\n}\n```\n\n2. Download by `Unity Package Manager` using git url `https://github.com/AkiKurisu/UniChat.git`\n\n## Core pipeline\n\n### Quick use\n\n1. Create or load\n\n```C#\npublic void CreatePipelineCtrl()\n{\n    //1. New chat model file (embedding database + text table + config)\n    ChatPipelineCtrl PipelineCtrl = new(new ChatModelFile() { fileName = $\"ChatModel_{Guid.NewGuid().ToString()[0..6]}\" });\n    //2. Load from filePath\n    PipelineCtrl= new(JsonConvert.DeserializeObject\u003cChatModelFile\u003e(File.ReadAllText(filePath)))\n}\n```\n\n2. Run the pipeline\n\n```C#\npublic bool RunPipeline()\n{\n    string input=\"Hello!\";\n    var context = await PipelineCtrl.RunPipeline(\"Hello!\");\n    if ((context.flag \u0026 (1 \u003c\u003c 1)) != 0)\n    {\n        // Get pipeline output\n        string output = context.CastStringValue();\n        // Update history\n        PipelineCtrl.History.AppendUserMessage(input);\n        PipelineCtrl.History.AppendBotMessage(output);\n        return true;\n    }\n}\n```\n\n3. Save the generated text and embedding vector\n\n```C#\npubic void Save()\n{\n    // PC save to {ApplicationPath}//UserData//{ModelName}\n    // Android save to {Application.persistentDataPath}//UserData//{ModelName}\n    PipelineCtrl.SaveModel();\n}\n```\n\n### Embedding model\n\nThe embedding model is used `BAAI/bge-small-zh-v1.5` by default and occupies the least video memory. It can be downloaded in Release, however only supports Chinese. You can download the same model from `HuggingFaceHub` and convert it to ONNX format.\n\nThe loading mode is optional `UserDataProvider`, `StreamingAssetsProvider` and `ResourcesProvider`, if installed `Unity.Addressables`, optional `AddressableProvider`.\n\nThe `UserDataProvider` file path is as follows:\n\n![UserData](./Docs/Images/userdata_provider.png)\n\n `ResourcesProvider` Place the files in the models folder in the Resources folder.\n\n `StreamingAssetsProvider` Place the files in the models folder in the StreamingAssets folder.\n\nAddress `AddressablesProvider` of is as follows:\n\n![Addressables](./Docs/Images/addressable-provider.png)\n\n## Chain\n\nUniChat is based on [LangChain C#](https://github.com/tryAGI/LangChain) using a chain structure to connect components in series.\n\nYou can see an sample in repo's Example.\n\nThe simple use is as follows:\n\n```C#\npublic class LLM_Chain_Example : MonoBehaviour\n{\n    public LLMSettingsAsset settingsAsset;\n    public AudioSource audioSource;\n    public async void Start()\n    {\n        var chatPrompt = @\"\n            You are an AI assistant that greets the world.\n            User: Hello!\n            Assistant:\";\n        var llm = LLMFactory.Create(LLMType.OpenAI, settingsAsset);\n        // Create chain\n        var chain =\n            Chain.Set(chatPrompt, outputKey: \"prompt\")\n            | Chain.LLM(llm, inputKey: \"prompt\", outputKey: \"chatResponse\");\n        // Run chain\n        string result = await chain.Run\u003cstring\u003e(\"chatResponse\");\n        Debug.Log(result);\n    }\n}\n```\n\n### Combined with the core pipeline\n\nThe above example uses `Chain` to call LLM directly, but to simplify searching the database and facilitate engineering, it is recommended to use `ChatPipelineCtrl` as the beginning of the chain.\n\nIf you run the following example, the first time you call LLM and the second time you reply directly from the database.\n\n\n```C#\npublic async void Start()\n{\n    // Create new chat model file with empty memory and embedding db\n    var chatModelFile = new ChatModelFile() { fileName = \"NewChatFile\", modelProvider = ModelProvider.AddressableProvider };\n    // Create an pipeline ctrl to run it\n    var pipelineCtrl = new ChatPipelineCtrl(chatModelFile, settingsAsset);\n    pipelineCtrl.SwitchGenerator(ChatGeneratorIds.ChatGPT);\n    // Init pipeline, set verbose to log status\n    await pipelineCtrl.InitializePipeline(new PipelineConfig { verbose = true });\n    // Add system prompt\n    pipelineCtrl.Memory.Context = \"You are my personal assistant, you should answer my questions.\";\n    // Create chain\n    var chain = pipelineCtrl.ToChain().Input(\"Hello assistant!\").CastStringValue(outputKey: \"text\");\n    // Run chain\n    string result = await chain.Run\u003cstring\u003e(\"text\");\n    // Save chat model\n    pipelineCtrl.SaveModel();\n}\n```\n\n### Stack Trace\n\nYou can trace the chain using the `Trace()` method, or add `UNICHAT_ALWAYS_TRACE_CHAIN` scripting symbol in Project Settings.\n\n\n| Method name | Return type | Description |\n| ------ | -------- | ---- |\n| `Trace(stackTrace, applyToContext)` |`void`|Trace chain|\n`stackTrace: bool`|| Enables stack tracing\n`applyToContext: bool` ||Applies to all subchains\n\n![StackTrace](./Docs/Images/stack-trace.png)\n\n\n\n## Middleware\n\n### Text to Speech\n\nIf you have a speech synthesis solution, you can refer to [VITSClient](./Runtime/Models/Audio/VITSClient.cs) the implementation of a TTS component📢.\n\nYou can use `AudioCache` to store speech so that it can be played when you pick up an answer from the database in offline mode.\n\n\n```C#\npublic class LLM_TTS_Chain_Example : MonoBehaviour\n{\n    public LLMSettingsAsset settingsAsset;\n    public AudioSource audioSource;\n    public async void Start()\n    {\n        // Create new chat model file with empty memory and embedding db\n        var chatModelFile = new ChatModelFile() { fileName = \"NewChatFile\", modelProvider = ModelProvider.AddressableProvider };\n        // Create an pipeline ctrl to run it\n        var pipelineCtrl = new ChatPipelineCtrl(chatModelFile, settingsAsset);\n        pipelineCtrl.SwitchGenerator(ChatGeneratorIds.OpenAI, true);\n        // Init pipeline, set verbose to log status\n        await pipelineCtrl.InitializePipeline(new PipelineConfig { verbose = true });\n        var vits = new VITSModel(lang: \"ja\");\n        // Add system prompt\n        pipelineCtrl.Memory.Context = \"You are my personal assistant, you should answer my questions.\";\n        // Create cache to cache audioClips and translated texts\n        var audioCache = AudioCache.CreateCache(chatModelFile.DirectoryPath);\n        var textCache = TextMemoryCache.CreateCache(chatModelFile.DirectoryPath);\n        // Create chain\n        var chain = pipelineCtrl.ToChain().Input(\"Hello assistant!\").CastStringValue(outputKey: \"text\")\n                                //Translate to japanese\n                                | Chain.Translate(new GoogleTranslator(\"en\", \"ja\")).UseCache(textCache)\n                                //Split them\n                                | Chain.Split(new RegexSplitter(@\"(?\u003c=[。！？! ?])\"), inputKey: \"translated_text\")\n                                //Auto batched\n                                | Chain.TTS(vits, inputKey: \"splitted_text\").UseCache(audioCache).Verbose(true);\n        // Run chain\n        (IReadOnlyList\u003cstring\u003e segments, IReadOnlyList\u003cAudioClip\u003e audioClips)\n            = await chain.Run\u003cIReadOnlyList\u003cstring\u003e, IReadOnlyList\u003cAudioClip\u003e\u003e(\"splitted_text\", \"audio\");\n        // Play audios\n        for (int i = 0; i \u003c audioClips.Count; ++i)\n        {\n            Debug.Log(segments[i]);\n            audioSource.clip = audioClips[i];\n            audioSource.Play();\n            await UniTask.WaitUntil(() =\u003e !audioSource.isPlaying);\n        }\n    }\n}\n```\n\n### Speech to Text\n\nYou can use a speech-to-text service such as [whisper.unity](https://github.com/Macoron/whisper.unity) for local inference🎤.\n\n```C#\npublic void RunSTTChain(AudioClip audioClip)\n{\n     WhisperModel whisperModel = await WhisperModel.FromPath(modelPath);\n     var chain = Chain.Set(audioClip, \"audio\")\n                         | Chain.STT(whisperModel, new WhisperSettings(){\n                             language=\"en\",\n                             initialPrompt=\"The following is a paragraph in English.\"\n                         });\n     Debug.Log(await chain.Run(\"text\"));\n}\n```\n\n### Sub-classifier\nYou can reduce the dependence on LLM by training a downstream classifier on the basis of the embedded model to complete some recognition tasks in the game (such as expression classifier🤗).\n\n**Notice**\n\n*1. You need to make the component in a Python environment.*\n\n*2. Currently, Sentis still requires you to manually export to ONNX format*\n\nBest practice: Use an embedded model to generate traits from your training data before training. Only the downstream model needs to be exported afterwards.\n\nThe following is an example `shape=(512,768,20)` of a multi-layer perceptron classifier with an export size of only 1.5MB:\n\n```python\nclass SubClassifier(nn.Module):\n    #input_dim is the output dim of your embedding model\n    def __init__(self, input_dim, hidden_dim, output_dim):\n        super(CustomClassifier, self).__init__()\n        \n        self.fc1 = nn.Linear(input_dim, hidden_dim)\n        self.relu = nn.ReLU()\n        self.dropout = nn.Dropout(p=0.1)\n        self.fc2 = nn.Linear(hidden_dim, output_dim)\n    \n    def forward(self, x):\n        x = self.fc1(x)\n        x = self.relu(x)\n        x = self.dropout(x)\n        x = self.fc2(x)\n        return x\n```\n\n### Tool Use\n\nInvoke tools based on ReActAgent workflow.\n\nHere is an example:\n\n```C#\nvar userCommand = @\"I want to watch a dance video.\";\nvar llm = LLMFactory.Create(LLMType.ChatGPT, settingsAsset) as OpenAIClient;\nllm.StopWords = new() { \"\\nObservation:\", \"\\n\\tObservation:\" };\n\n// Create agent with muti-tools\nvar chain =\n    Chain.Set(userCommand)\n    | Chain.ReActAgentExecutor(llm)\n        .UseTool(new AgentLambdaTool(\n            \"Play random dance video\",\n            @\"A wrapper to select random dance video and play it. Input should be 'None'.\",\n            (e) =\u003e\n            {\n                PlayRandomDanceVideo();\n                //Notice agent it finished its work\n                return UniTask.FromResult(\"Dance video 'Queencard' is playing now.\");\n            }))\n        .UseTool(new AgentLambdaTool(\n            \"Sleep\",\n            @\"A wrapper to sleep.\",\n            (e) =\u003e\n            {\n                return UniTask.FromResult(\"You are now sleeping.\");\n            }))\n        .Verbose(true);\n\n// Run chain\nDebug.Log(await chain.Run(\"text\"));\n```\n\n## Demo\n\nHere are some apps I've made. Since they include some commercial plugins, only Build versions are available.\n\nSee [Release](https://github.com/AkiKurisu/UniChat/releases) page\n\n\nBased on UniChat to make a similar application in Unity\n\u003e The synchronized repository version is `V0.0.1-alpha`, the Demo is waiting to be updated.\n\n### Download\n\n![Chat-View](./Docs/Images/chat-view.png)\n\nSee [Release](https://github.com/AkiKurisu/UniChat/releases) page\n\n### Features\n\n#### Personalization: Character Cards\n\nDemo uses `TavernAI` the character data structure, and we can write the character's personality, sample conversations, and chat scenarios into pictures.\n\n![Setting View](./Docs/Images/setting-view.png)\n\nIf you use `TavernAI` a character card, the cue word above is overwritten.\n\n## More Showcase\n\nUsing UniChat to build a galgame with ai character.\n\n![Showcase](./Docs/Images/aichat-example.png)\n\u003e Character model from @Illusion.\n\n## Quote\n\n### Blogs\n- Make a ChatBox in Unity\n    \u003ehttps://www.akikurisu.com/blog/posts/create-chatbox-in-unity-2024-03-19/\n- Using NLP Natural Language Processing Technology in Unity\n    \u003ehttps://www.akikurisu.com/blog/posts/use-nlp-in-unity-2024-04-03/\n\n### Repo\n- https://github.com/langchain-ai/langchain\n- https://github.com/tryAGI/LangChain\n\n### Paper\n- Yao S, Zhao J, Yu D, et al. React: Synergizing reasoning and acting in language models[J]. arXiv preprint arXiv:2210.03629, 2022.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakikurisu%2Funichat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fakikurisu%2Funichat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakikurisu%2Funichat/lists"}