{"id":13560260,"url":"https://github.com/mehrandvd/skunit","last_synced_at":"2025-04-05T06:04:44.260Z","repository":{"id":213426603,"uuid":"732945870","full_name":"mehrandvd/skunit","owner":"mehrandvd","description":"skUnit is a testing tool for .NET AI units, such as IChatClient and SK kernels.","archived":false,"fork":false,"pushed_at":"2025-03-16T08:19:25.000Z","size":286,"stargazers_count":158,"open_issues_count":4,"forks_count":17,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-29T05:02:00.726Z","etag":null,"topics":["csharp","dotnet","gpt-plugins","openai","semantic-kernel","test","test-automation","testing","testing-tools"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mehrandvd.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-18T08:12:07.000Z","updated_at":"2025-03-15T04:00:14.000Z","dependencies_parsed_at":"2024-01-13T21:21:50.782Z","dependency_job_id":"4f115db2-1211-4505-a333-f4de1568f86b","html_url":"https://github.com/mehrandvd/skunit","commit_stats":{"total_commits":111,"total_committers":2,"mean_commits":55.5,"dds":"0.29729729729729726","last_synced_commit":"f48387d7cf5fff2d595ea3936cbc2633c410c4ec"},"previous_names":["mehrandvd/skunit"],"tags_count":39,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mehrandvd%2Fskunit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mehrandvd%2Fskunit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mehrandvd%2Fskunit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mehrandvd%2Fskunit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mehrandvd","download_url":"https://codeload.github.com/mehrandvd/skunit/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247294537,"owners_count":20915340,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csharp","dotnet","gpt-plugins","openai","semantic-kernel","test","test-automation","testing","testing-tools"],"created_at":"2024-08-01T13:00:40.458Z","updated_at":"2025-04-05T06:04:44.234Z","avatar_url":"https://github.com/mehrandvd.png","language":"C#","funding_links":[],"categories":["Semantickernel Framework","📚 Projects (1974 total)"],"sub_categories":["MCP Servers"],"readme":"# skUnit\n[![Build and Deploy](https://github.com/mehrandvd/skUnit/actions/workflows/build.yml/badge.svg)](https://github.com/mehrandvd/skUnit/actions/workflows/build.yml)\n[![NuGet version (skUnit)](https://img.shields.io/nuget/v/skUnit.svg?style=flat)](https://www.nuget.org/packages/skUnit/)\n[![NuGet downloads](https://img.shields.io/nuget/dt/skUnit.svg?style=flat)](https://www.nuget.org/packages/skUnit)\n\n**skUnit** is a testing tool for any `IChatClient` and [SemanticKernel](https://github.com/microsoft/semantic-kernel) units, such as _kernels_, _chat services_ and ...\n\nYou can write [**Chat Scenarios**](https://github.com/mehrandvd/skunit/blob/main/docs/chat-scenario-spec.md), which test a sequence of interactions between the user and an `IChatClient` or a SemanticKernel.\n\n# Chat Scenarios\n\nA chat scenario is a way of testing how an `IChatClient`, responds to user inputs in skUnit. \nA chat scenario consists of one or more sub-scenarios, each representing a dialogue turn between the user and the agent.\n\n## Example\nThis is an example of a chat scenario with two sub-scenarios:\n\n```md\n# SCENARIO Height Discussion\n\n## [USER]\nIs Eiffel tall?\n\n## [AGENT]\nYes it is\n\n### CHECK SemanticCondition\nIt agrees that the Eiffel Tower is tall or expresses a positive sentiment.\n\n## [USER]\nWhat about Everest Mountain?\n\n## [AGENT]\nYes it is tall too\n\n### CHECK SemanticCondition\nIt agrees that Everest mountain is tall or expresses a positive sentiment.\n```\n\n![image](https://github.com/mehrandvd/skunit/assets/5070766/156b0831-e4f3-4e4b-b1b0-e2ec868efb5f)\n\n### Sub-scenario 1\nThe first sub-scenario tests how the agent responds to the question `Is Eiffel tall?`. \nThe expected answer is something like `Yes it is`, but this is not an exact match. It is just a guideline for the desired response.\n\nWhen the scenario is executed, the OpenAI generates an actual answer, such as `Yes it is quite tall.`. \nThe next statement `CHECK SemanticCondition` is an assertion that verifies if the actual answer meets the specified condition: \n`It agrees that the Eiffel Tower is tall or expresses a positive sentiment.`\n\n### Sub-scenario 2\nThe second sub-scenario tests how the agent responds to the follow-up question `What about Everest mountain?`. \nThe expected answer is something like `Yes it is tall too`, but again, this is not an exact match. It is just a guideline for the desired response.\n\nWhen the scenario is executed, the OpenAI generates an actual answer, such as `Yes it is very tall indeed.`. \nThe next statement `CHECK SemanticCondition` is an assertion that verifies if the actual answer meets the specified condition: \n`It agrees that Everest mountain is tall or expresses a positive sentiment.`\n\nAs you can see, this sub-scenario does not depend on the exact wording of the previous answer. \nIt assumes that the agent responded in the expected way and continues the test. \nThis makes writing long tests easier, as you can rely on the agent's answers to design your test. \nOtherwise, you would have to account for different variations of the intermediate answers every time you run the test.\n\nHowever, `SemanticSimilar` is not the only assertion method. There are many more assertion checks available (like **SemanticCondition**, **Equals**). \n\nYou can see the full list of CHECK statements here: [CHECK Statement spec](https://github.com/mehrandvd/skunit/blob/main/docs/check-statements-spec.md).\n\n## Scenarios are Valid Markdowns\n\nOne of the benefits of skUnit scenarios is that they are valid **Markdown** files, which makes them very readable and easy to edit. \n\n\u003e skUnit scenarios are valid **Markdown** files, which makes them very readable and easy to edit.\n\nFor example, you can see how clear and simple this scenario is: [Chatting about Eiffel height](https://github.com/mehrandvd/skunit/blob/main/src/skUnit.Tests/SemanticKernelTests/ChatScenarioTests/Samples/EiffelTallChat/skchat.md)\n\n![image](https://github.com/mehrandvd/skunit/assets/5070766/53d009a9-4a0b-44dc-91e0-b0be81b4c5a7)\n\n## Executing a Test Using a Scenario\nExecuting tests is a straightforward process. You have the flexibility to utilize any preferred test frameworks such as xUnit, nUnit, or MSTest. With just two lines of code, you can load and run a test:\n\n```csharp\nvar markdown = // Load it from .md file\nvar scenarios = await ChatScenario.LoadFromText(markdown);\nvar chatClient = CreateChatClient();\nawait ScenarioAssert.PassAsync(scenarios, chatClient);\n\n// Or configure your way of processing the chat.\nawait ScenarioAssert.PassAsync(scenarios,\n  getAnswerFunc: async history =\u003e\n            {\n                var result = // your logic to be tested;\n                return result;\n            });\n```\n\nThe test output will be generated incrementally, line by line:\n\n```md\n# SCENARIO Height Discussion\n\n## [USER]\nIs Eiffel tall?\n\n## [EXPECTED ANSWER]\nYes it is\n\n### [ACTUAL ANSWER]\nYes, the Eiffel Tower in Paris, France, is tall at 330 meters (1,083 feet) in height.\n\n### CHECK Condition\nConfirms that the Eiffel Tower is tall or expresses positivity.\n✅ OK\n\n## [USER]\nWhat about Everest Mountain?\n\n## [EXPECTED ANSWER]\nYes it is tall too\n\n### [ACTUAL ANSWER]\nYes, Mount Everest is the tallest mountain in the world, with a peak that reaches 29,032 feet (8,849 meters) above sea level.\n\n### CHECK Condition\nThe sentence is positive.\n✅ OK\n\n## [USER]\nWhat about a mouse?\n\n## [EXPECTED ANSWER]\nNo, it is not tall.\n\n### [ACTUAL ANSWER]\nNo, a mouse is not tall.\n\n### CHECK Condition\nThe sentence is negative.\n✅ OK\n\n## [USER]\nGive me a JSON containing the Eiffel height.\nExample: \n{\n\t\"height\": \"330 meters\"\n}\n\n## [EXPECTED ANSWER]\n{\n\t\"height\": \"330 meters\"\n}\n\n### [ACTUAL ANSWER]\n{\n\t\"height\": \"330 meters\"\n}\n\n### CHECK JsonCheck\n{\n\t\"height\": [\"NotEmpty\", \"\"]\n}\n✅ OK\n\n### CHECK JsonCheck\n{\n\t\"height\": [\"Contain\", \"meters\"]\n}\n✅ OK\n```\n\nThis output is generated line by line as the test is executed:\n\n![image](https://github.com/mehrandvd/skunit/assets/5070766/f3ef8a37-ceab-444f-b6f4-098557b61bfa)\n\n\n## Documents\nTo better understand skUnit, Check these documents:\n - [Chat Scenario Spec](https://github.com/mehrandvd/skunit/blob/main/docs/chat-scenario-spec.md): The details of writing an ChatScenario.\n - [CHECK Statement Spec](https://github.com/mehrandvd/skunit/blob/main/docs/check-statements-spec.md): The various `CHECK` statements that you can use for assertion.\n\n## Requirements\n- .NET 7.0 or higher\n- An OpenAI API key\n\n## Installation\nYou can easily add **skUnit** to your project as it is available as a [NuGet](https://www.nuget.org/packages/skUnit) package. To install it, execute the following command in your terminal:\n```bash\ndotnet add package skUnit\n```\n\nAfterwards, you'll need to instantiate the `SemanticKernelAssert` class in your test constructor. This requires passing your OpenAI subscription details as parameters:\n```csharp\npublic class MyTest\n{\n  ScenarioAssert ScenarioAssert { get; set; }\n  MyTest(ITestOutputHelper output)\n  {\n    ScenarioAssert = new ScenarioAssert(\n      new AzureOpenAIClient(...),\n      output.WriteLine);\n  }\n\n  [Fact]\n  TestChat()\n  {\n    // Arrange\n    var chatClient = // Build your IChatClient\n    string markdown = // Load your markdown scenario from a .md file.\n    var scenarios = await ChatScenario.LoadFromTest(markdown);\n\n    // Action + Assert\n    await ScenarioAssert.PassAsync(scenarios, chatClient);\n  }\n}\n```\nAnd that's all there is to it! 😊\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmehrandvd%2Fskunit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmehrandvd%2Fskunit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmehrandvd%2Fskunit/lists"}