{"id":37054080,"url":"https://github.com/sa-es-ir/detester","last_synced_at":"2026-01-14T06:06:19.416Z","repository":{"id":318375532,"uuid":"1070084048","full_name":"sa-es-ir/detester","owner":"sa-es-ir","description":"AI Deterministic Tester","archived":false,"fork":false,"pushed_at":"2025-11-30T20:58:09.000Z","size":521,"stargazers_count":22,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-12-01T13:14:39.102Z","etag":null,"topics":["ai","csharp","deterministic","deterministic-ai","testing"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sa-es-ir.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-05T08:26:04.000Z","updated_at":"2025-12-01T12:21:23.000Z","dependencies_parsed_at":"2025-10-06T21:19:32.064Z","dependency_job_id":"0ed38c92-ec3a-4e8f-9d13-e7b0aba2cba3","html_url":"https://github.com/sa-es-ir/detester","commit_stats":null,"previous_names":["sa-es-ir/detester"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/sa-es-ir/detester","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sa-es-ir%2Fdetester","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sa-es-ir%2Fdetester/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sa-es-ir%2Fdetester/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sa-es-ir%2Fdetester/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sa-es-ir","download_url":"https://codeload.github.com/sa-es-ir/detester/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sa-es-ir%2Fdetester/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28412187,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T05:26:33.345Z","status":"ssl_error","status_checked_at":"2026-01-14T05:21:57.251Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","csharp","deterministic","deterministic-ai","testing"],"created_at":"2026-01-14T06:06:18.718Z","updated_at":"2026-01-14T06:06:19.410Z","avatar_url":"https://github.com/sa-es-ir.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\u003cimg src=\"detester_logo.png\" width=\"300\"\u003e\u003c/p\u003e\n\n\u003cdiv align=\"center\"\u003e\n\n[![NuGet](https://img.shields.io/nuget/v/Detester.svg)](https://www.nuget.org/packages/Detester/)\n[![GitHub stars](https://img.shields.io/github/stars/sa-es-ir/Detester.svg)](https://github.com/sa-es-ir/Detester/stargazers)\n\u003c/div\u003e\n\n# Detester\n\nDetester is a .NET library that enables you to write deterministic tests for AI-powered applications. It provides a fluent builder API for testing AI responses, ensuring consistency and reliability in your AI integrations.\n\n## Features\n\n- **Fluent Builder API**: Chain multiple prompts and assertions in a readable, intuitive way\n- **Any AI Provider Support**: Works with any `IChatClient` implementation (OpenAI, Azure OpenAI, Ollama, etc.)\n- **Model Instructions**: Set system messages to guide model behavior and responses\n- **Response Validation**: Assert that AI responses contain expected keywords or text\n- **Function/Tool Call Verification**: Verify that AI models call the correct functions with expected parameters\n- **JSON Response Validation**: Deserialize and validate JSON responses from AI models with type-safe validation\n- **Method Chaining**: Combine multiple prompts and assertions in a single test flow\n- **Extensible**: Build on Microsoft.Extensions.AI abstractions for maximum flexibility\n\n## Installation\n\n```bash\ndotnet add package Detester\n```\n\nFor OpenAI support, also install:\n```bash\ndotnet add package Microsoft.Extensions.AI.OpenAI\n```\n\nFor Azure OpenAI support, also install:\n```bash\ndotnet add package Azure.AI.OpenAI\ndotnet add package Microsoft.Extensions.AI.OpenAI\n```\n\n## Quick Start\n\n### Using OpenAI\n\n```csharp\nusing Detester;\nusing Microsoft.Extensions.AI;\nusing OpenAI;\n\n// Create OpenAI client and wrap it as IChatClient\nvar openAIClient = new OpenAIClient(\"your-openai-api-key\");\nvar chatClient = openAIClient.GetChatClient(\"gpt-4\").AsIChatClient();\n\n// Create a builder with the chat client\nvar builder = DetesterFactory.Create(chatClient);\n\n// Execute a test\nawait builder\n    .WithPrompt(\"What is the capital of France?\")\n    .ShouldContainResponse(\"Paris\")\n    .AssertAsync();\n```\n\n### Using Azure OpenAI\n\n```csharp\nusing Azure.AI.OpenAI;\nusing Detester;\nusing Microsoft.Extensions.AI;\nusing System.ClientModel;\n\n// Create Azure OpenAI client and wrap it as IChatClient\nvar azureClient = new AzureOpenAIClient(\n    new Uri(\"https://your-resource.openai.azure.com\"),\n    new ApiKeyCredential(\"your-azure-api-key\"));\nvar chatClient = azureClient.GetChatClient(\"gpt-4-deployment\").AsIChatClient();\n\n// Create a builder with the chat client\nvar builder = DetesterFactory.Create(chatClient);\n\n// Execute a test\nawait builder\n    .WithPrompt(\"Explain quantum computing in simple terms\")\n    .ShouldContainResponse(\"quantum\")\n    .AssertAsync();\n```\n\n### Using Any IChatClient\n\nDetester works with any `IChatClient` implementation from `Microsoft.Extensions.AI`:\n\n```csharp\nusing Detester;\nusing Microsoft.Extensions.AI;\n\n// Use any IChatClient implementation (OpenAI, Azure OpenAI, Ollama, custom, etc.)\nIChatClient chatClient = // your chat client implementation\nvar builder = DetesterFactory.Create(chatClient);\n\nawait builder\n    .WithPrompt(\"Test prompt\")\n    .ShouldContainResponse(\"expected text\")\n    .AssertAsync();\n```\n\n## Advanced Usage\n\n### Setting Model Instructions\n\nSet custom instructions (system messages) to guide the model's behavior:\n\n```csharp\nawait builder\n    .WithInstruction(\"You are a helpful assistant that provides concise answers.\")\n    .WithPrompt(\"What is machine learning?\")\n    .ShouldContainResponse(\"algorithm\")\n    .AssertAsync();\n```\n\nInstructions are sent as system messages before any prompts, allowing you to control the model's tone, style, and behavior throughout the conversation:\n\n```csharp\nawait builder\n    .WithInstruction(\"You are a Python expert. Always provide code examples.\")\n    .WithPrompt(\"How do I read a file in Python?\")\n    .ShouldContainResponse(\"open(\")\n    .ShouldContainResponse(\"read(\")\n    .AssertAsync();\n```\n\n### Multiple Prompts\n\nTest conversational flows by chaining multiple prompts:\n\n```csharp\nawait builder\n    .WithPrompt(\"Hello, I need help with coding\")\n    .WithPrompt(\"Can you explain what a variable is?\")\n    .ShouldContainResponse(\"variable\")\n    .AssertAsync();\n```\n\n### Multiple Assertions\n\nAdd multiple response checks:\n\n```csharp\nawait builder\n    .WithPrompt(\"Write a haiku about programming\")\n    .ShouldContainResponse(\"code\")\n    .ShouldContainResponse(\"lines\")\n    .AssertAsync();\n```\n\n### Batch Prompts\n\nAdd multiple prompts at once:\n\n```csharp\nawait builder\n    .WithPrompts(\n        \"What is machine learning?\",\n        \"How does it differ from traditional programming?\",\n        \"Give me a practical example\")\n    .ShouldContainResponse(\"algorithm\")\n    .ShouldContainResponse(\"data\")\n    .AssertAsync();\n```\n\n### OR Assertions\n\nUse `OrShouldContainResponse` to create flexible response validation where at least one of the alternatives must match:\n\n```csharp\nawait builder\n    .WithPrompt(\"What is the capital of France?\")\n    .ShouldContainResponse(\"capital\")\n    .OrShouldContainResponse(\"city\")\n    .OrShouldContainResponse(\"Paris\")\n    .AssertAsync();\n```\n\nIn this example, the test passes if the response contains \"capital\" OR \"city\" OR \"Paris\". You can chain multiple OR conditions, and the test will pass if any one of them is found in the response.\n\n#### Combining AND and OR Assertions\n\nYou can mix `ShouldContainResponse` (AND) with `OrShouldContainResponse` (OR) for complex validation:\n\n```csharp\nawait builder\n    .WithPrompt(\"Explain machine learning\")\n    .ShouldContainResponse(\"algorithm\")  // Must contain \"algorithm\"\n    .ShouldContainResponse(\"data\")       // AND must contain \"data\"\n    .OrShouldContainResponse(\"train\")    // AND must contain \"train\" OR \"data\"\n    .AssertAsync();\n```\n\nNote: `OrShouldContainResponse` creates an OR group with the immediately preceding assertion. Each subsequent `OrShouldContainResponse` adds another alternative to that OR group.\n\n### Function/Tool Call Verification\n\nDetester supports verifying that AI models call the correct functions/tools with expected parameters. This is useful for testing AI applications that use function calling capabilities.\n\n#### Basic Function Call Verification\n\nVerify that a specific function is called:\n\n```csharp\nawait builder\n    .WithPrompt(\"What's the weather in Paris?\")\n    .ShouldCallFunction(\"get_weather\")\n    .AssertAsync();\n```\n\n#### Verify Function Parameters\n\nCheck that functions are called with the correct parameters:\n\n```csharp\nawait builder\n    .WithPrompt(\"What's the weather in Paris in celsius?\")\n    .ShouldCallFunctionWithParameters(\"get_weather\", \n        new Dictionary\u003cstring, object?\u003e \n        { \n            { \"location\", \"Paris\" },\n            { \"units\", \"celsius\" }\n        })\n    .AssertAsync();\n```\n\n#### Multiple Function Calls\n\nVerify multiple function calls in a single response:\n\n```csharp\nawait builder\n    .WithPrompt(\"Compare the weather in Paris and London\")\n    .ShouldCallFunction(\"get_weather\")\n    .ShouldCallFunction(\"get_weather\")\n    .AssertAsync();\n```\n\n#### Combined Verification\n\nCombine function call verification with text response assertions:\n\n```csharp\nawait builder\n    .WithPrompt(\"What's the capital of France?\")\n    .ShouldCallFunction(\"get_capital\")\n    .ShouldContainResponse(\"Paris\")\n    .AssertAsync();\n```\n\nFor more detailed information and examples, see the [Function Calling Guide](docs/function-calling-guide.md).\n\n### JSON Response Validation\n\nDetester supports validating JSON responses from AI models by deserializing them to C# types and optionally validating the deserialized objects. This is useful for testing structured outputs from language models.\n\n#### Basic JSON Validation\n\nVerify that the response can be deserialized to a specific type:\n\n```csharp\npublic class User\n{\n    public string? FirstName { get; set; }\n    public string? LastName { get; set; }\n    public int Age { get; set; }\n    public DateTime JoinDate { get; set; }\n}\n\nawait builder\n    .WithPrompt(\"Who is the last user joined?\")\n    .ShouldHaveJsonOfType\u003cUser\u003e(new JsonSerializerOptions { PropertyNameCaseInsensitive = true })\n    .AssertAsync();\n```\n\n#### JSON Validation with Custom Validation\n\nAdd custom validation logic to verify the deserialized object:\n\n```csharp\nawait builder\n    .WithPrompt(\"Who is the last user joined?\")\n    .ShouldHaveJsonOfType\u003cUser\u003e(\n        new JsonSerializerOptions { PropertyNameCaseInsensitive = true },\n        user =\u003e user.Age \u003e 30 \u0026\u0026 user.FirstName!.Contains(\"Jo\"))\n    .AssertAsync();\n```\n\n#### Complex JSON Validation\n\nCombine multiple validations:\n\n```csharp\nawait builder\n    .WithPrompt(\"Get user details\")\n    .ShouldContainResponse(\"Joe\")  // Text assertion\n    .ShouldHaveJsonOfType\u003cUser\u003e(\n        new JsonSerializerOptions { PropertyNameCaseInsensitive = true },\n        user =\u003e user.Age \u003e 18)  // JSON validation\n    .ShouldHaveJsonOfType\u003cUser\u003e(\n        new JsonSerializerOptions { PropertyNameCaseInsensitive = true },\n        user =\u003e user.LastName == \"Doe\")  // Additional JSON validation\n    .AssertAsync();\n```\n\n**Note:** \n- The JSON validation uses `System.Text.Json` for deserialization\n- Deserialization exceptions are caught and wrapped in `DetesterException` with helpful error messages\n- If validation fails, the test throws `DetesterException` with details about what went wrong\n- For case-insensitive property name matching, use `JsonSerializerOptions { PropertyNameCaseInsensitive = true }`\n\n## Testing Example with xUnit\n\n```csharp\nusing Detester;\nusing Microsoft.Extensions.AI;\nusing OpenAI;\n\npublic class AITests\n{\n    [Fact]\n    public async Task TestAIResponse()\n    {\n        // Arrange - Create your chat client\n        var openAIClient = new OpenAIClient(\n            Environment.GetEnvironmentVariable(\"OPENAI_API_KEY\")!);\n        var chatClient = openAIClient.GetChatClient(\"gpt-4\").AsIChatClient();\n        var builder = DetesterFactory.Create(chatClient);\n\n        // Act \u0026 Assert\n        await builder\n            .WithPrompt(\"What is 2+2?\")\n            .ShouldContainResponse(\"4\")\n            .AssertAsync();\n    }\n}\n```\n\n## Configuration\n\nDetester uses `IChatClient` from `Microsoft.Extensions.AI` as its core abstraction. You create and configure your chat client according to the provider's documentation:\n\n### OpenAI\n\n```csharp\n// Install: dotnet add package Microsoft.Extensions.AI.OpenAI\nvar openAIClient = new OpenAIClient(Environment.GetEnvironmentVariable(\"OPENAI_API_KEY\")!);\nvar chatClient = openAIClient.GetChatClient(\"gpt-4\").AsIChatClient();\n```\n\n### Azure OpenAI\n\n```csharp\n// Install: dotnet add package Azure.AI.OpenAI\n// Install: dotnet add package Microsoft.Extensions.AI.OpenAI\nvar azureClient = new AzureOpenAIClient(\n    new Uri(Environment.GetEnvironmentVariable(\"AZURE_OPENAI_ENDPOINT\")!),\n    new ApiKeyCredential(Environment.GetEnvironmentVariable(\"AZURE_OPENAI_API_KEY\")!));\nvar chatClient = azureClient.GetChatClient(\"your-deployment-name\").AsIChatClient();\n```\n\n## API Reference\n\n### DetesterFactory\n\n- `Create(chatClient)`: Create a builder with an IChatClient implementation\n\n### IDetesterBuilder\n\n- `WithInstruction(instruction)`: Set the instruction (system message) for the AI model\n- `WithPrompt(prompt)`: Add a single prompt\n- `WithPrompts(params prompts)`: Add multiple prompts\n- `ShouldContainResponse(expectedText)`: Assert response contains text (case-insensitive, AND condition)\n- `OrShouldContainResponse(expectedText)`: Assert response contains alternative text (case-insensitive, OR condition)\n- `ShouldCallFunction(functionName)`: Assert that a specific function/tool was called\n- `ShouldCallFunctionWithParameters(functionName, parameters)`: Assert that a function was called with specific parameters\n- `ShouldHaveJsonOfType\u003cT\u003e(options, validator)`: Assert that response contains valid JSON deserializable to type T, with optional validation\n- `AssertAsync(cancellationToken)`: Assert the test by executing prompts and validating responses\n\n## Error Handling\n\nDetester throws `DetesterException` when:\n- No prompts are provided before execution\n- Expected text is not found in the response\n- None of the OR alternatives are found in the response\n- Expected function call is not found\n- JSON deserialization fails or validation fails\n\nDetester throws `InvalidOperationException` when:\n- `OrShouldContainResponse` is called without a prior assertion\n\nExample:\n\n```csharp\ntry\n{\n    await builder\n        .WithPrompt(\"What is AI?\")\n        .ShouldContainResponse(\"impossible text that won't appear\")\n        .AssertAsync();\n}\ncatch (DetesterException ex)\n{\n    Console.WriteLine($\"Test failed: {ex.Message}\");\n}\n```\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## License\n\nThis project is licensed under the MIT License.\n\n## Acknowledgments\n\nBuilt on top of [Microsoft.Extensions.AI](https://www.nuget.org/packages/Microsoft.Extensions.AI) for seamless integration with AI services.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsa-es-ir%2Fdetester","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsa-es-ir%2Fdetester","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsa-es-ir%2Fdetester/lists"}