{"id":17950783,"url":"https://github.com/jonathanpeppers/mypal","last_synced_at":"2025-03-25T00:31:24.455Z","repository":{"id":251397456,"uuid":"835303294","full_name":"jonathanpeppers/MyPal","owner":"jonathanpeppers","description":"A .NET MAUI sample application that uses omni-input LLMs to create \"My Pal\", a friendly koala that insults you based on a selfie!","archived":false,"fork":false,"pushed_at":"2024-10-28T14:32:08.000Z","size":26925,"stargazers_count":4,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-16T01:33:46.259Z","etag":null,"topics":["csharp","dotnet","llms","maui","openai"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jonathanpeppers.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-29T14:59:47.000Z","updated_at":"2025-01-23T20:40:34.000Z","dependencies_parsed_at":"2024-10-26T00:17:05.322Z","dependency_job_id":null,"html_url":"https://github.com/jonathanpeppers/MyPal","commit_stats":null,"previous_names":["jonathanpeppers/mypal"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonathanpeppers%2FMyPal","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonathanpeppers%2FMyPal/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonathanpeppers%2FMyPal/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonathanpeppers%2FMyPal/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jonathanpeppers","download_url":"https://codeload.github.com/jonathanpeppers/MyPal/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245377919,"owners_count":20605374,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csharp","dotnet","llms","maui","openai"],"created_at":"2024-10-29T09:40:29.533Z","updated_at":"2025-03-25T00:31:22.842Z","avatar_url":"https://github.com/jonathanpeppers.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MyPal\n\nA .NET MAUI sample application that uses omni-input LLMs to create \"My\nPal\", a friendly koala that insults you based on a selfie!\n\n![Screenshot of MyPal application](assets/screenshot.png)\n\nhttps://github.com/user-attachments/assets/a58c7494-07f2-42d6-b639-f1e66fecc59e\n\nThis sample runs on iOS and Android, but could be adapted for desktop\nin the future.\n\n## Azure Setup\n\nThis app requires 3 Azure services:\n\n* Azure OpenAI: `gpt-4o` instance\n* Azure OpenAI: `tts` instance\n* Azure Application Insights: for telemetry\n\nI created these in North Central US, as that region supports both gpt-4o and tts:\n\n* https://learn.microsoft.com/azure/ai-services/openai/concepts/models\n\nWhen creating a model deployment in Azure OpenAI Studio, name them\nexactly `gpt-4o` and `tts`, otherwise you will need to use different\nnames in `MyPalWebClient.cs`.\n\nYou can also try a `tts-hd` model, but that seemed to just increase\nthe latency of each request. It would be more useful for making\nhigh-quality voiceovers, though.\n\nLastly, I made an Azure Log Analytics (dependency of Application\nInsights) and Azure Application Insights instance in North Central US.\nIf you want to skip this step, you can remove the Application Insights\ncode from the .NET MAUI app.\n\n## Secrets\n\nFor any of the three apps to work:\n\n* `MyPal.Benchmarks`\n* `MyPal.Console`\n* `MyPal.MauiApp`\n\nCreate a `runtimeconfig.template.json` with the contents:\n\n```json\n{\n  \"configProperties\": {\n    \"APPLICATIONINSIGHTS_CONNECTION_STRING\": \"secret here\",\n    \"OPENAI_API_KEY\": \"secret here\"\n  }\n}\n```\n\n`APPLICATIONINSIGHTS_CONNECTION_STRING` is only required for the .NET MAUI app.\n\n## Notes about Performance\n\nOriginally, I made two requests:\n\n* `CompleteChatAsync()` from `gpt-4o`, returns text. Around 11 seconds!\n* `GenerateSpeechFromTextAsync()` from `tts-hd`, returns audio. Around 5 seconds!\n\nSome initial performance changes, I made:\n\n* Use a 150x200 image, which seems to be good enough in detail for\n  `gpt-4o` to read as input. Down to ~10 second request.\n\n* Use `tts` instead of `tts-hd`, it is faster but lower quality. Down\n  to a ~4 second request.\n\nThis is still about 14 seconds, which is too slow for a real-time\n\"pal\" like I was trying to make.\n\nNext, I tried the streaming API for the initial request:\n\n* Call `CompleteChatStreamingAsync()` from `gpt-4o`.\n\n* When the first complete sentence is returned, start the audio\n  generation with `GenerateSpeechFromTextAsync()` from `tts`.\n\nThis made me track a new metric, \"time to first audio\", which is\naround 9 seconds. This is still too slow for a real-time \"pal\", but\nI'm going with this implementation for now.\n\nThe demos we see in ChatGPT videos are likely using a more powerful\nAPIs we don't have access to yet:\n\n* https://www.youtube.com/live/DQacCB9tDaw\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonathanpeppers%2Fmypal","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjonathanpeppers%2Fmypal","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonathanpeppers%2Fmypal/lists"}