{"id":37071206,"url":"https://github.com/chigwell/llmtestr","last_synced_at":"2026-01-14T08:19:21.874Z","repository":{"id":329749779,"uuid":"1120570836","full_name":"chigwell/llmtestr","owner":"chigwell","description":"A new package that helps developers integration-test AI and LLM applications by validating structured outputs. It takes a user's test scenario or prompt as input, sends it to an LLM, and uses pattern ","archived":false,"fork":false,"pushed_at":"2025-12-21T13:56:47.000Z","size":4,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-23T04:56:06.136Z","etag":null,"topics":["ai-powered-system-testing","ai-testing","automated-test-execution","code-snippet-validation","developer-tooling","formatting-error-detection","integration-testing","json-schema-validation","llm-validation","output-consistency-enforcement","output-content-verification","pattern-matching","prompt-driven-testing","regression-detection","response-format-checking","schema-enforcement","structured-output-verification","tagged-response-validation","test-automation","test-scenario-input"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/llmtestr/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chigwell.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-21T13:56:40.000Z","updated_at":"2025-12-21T13:57:02.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/chigwell/llmtestr","commit_stats":null,"previous_names":["chigwell/llmtestr"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/chigwell/llmtestr","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chigwell%2Fllmtestr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chigwell%2Fllmtestr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chigwell%2Fllmtestr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chigwell%2Fllmtestr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chigwell","download_url":"https://codeload.github.com/chigwell/llmtestr/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chigwell%2Fllmtestr/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28413765,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T08:16:59.381Z","status":"ssl_error","status_checked_at":"2026-01-14T08:13:45.490Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-powered-system-testing","ai-testing","automated-test-execution","code-snippet-validation","developer-tooling","formatting-error-detection","integration-testing","json-schema-validation","llm-validation","output-consistency-enforcement","output-content-verification","pattern-matching","prompt-driven-testing","regression-detection","response-format-checking","schema-enforcement","structured-output-verification","tagged-response-validation","test-automation","test-scenario-input"],"created_at":"2026-01-14T08:19:21.047Z","updated_at":"2026-01-14T08:19:21.865Z","avatar_url":"https://github.com/chigwell.png","language":"Python","readme":"# LLM Test Helper (`llmtestr`)\n[![PyPI version](https://badge.fury.io/py/llmtestr.svg)](https://badge.fury.io/py/llmtestr)\n[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)\n[![Downloads](https://static.pepy.tech/badge/llmtestr)](https://pepy.tech/project/llmtestr)\n[![LinkedIn](https://img.shields.io/badge/LinkedIn-blue)](https://www.linkedin.com/in/eugene-evstafev-716669181/)\n\n\n`llmtestr` is a Python package designed to assist developers in integration-testing AI and Language Model applications by validating structured outputs. It provides a simple interface to send a prompt or test scenario to an LLM, then verifies that the response matches predefined patterns using pattern matching mechanisms. This helps ensure that your LLM outputs adhere to expected formats such as code snippets, JSON structures, or tagged responses, making it easier to catch formatting errors, regressions, or inconsistencies during development and testing.\n\n## Installation\n\nInstall the package via pip:\n\n```bash\npip install llmtestr\n```\n\n## Usage\n\nHere's a basic example of how to use `llmtestr` in your Python code:\n\n```python\nfrom llmtestr import llmtestr\n\nresponse = llmtestr(user_input=\"Your test prompt here\")\nprint(response)\n```\n\n## Function Parameters\n\n- `user_input` (*str*): The prompt or test scenario you want to evaluate.\n- `llm` (*Optional[BaseChatModel]*): An optional `langchain` LLM instance to use. If not provided, `llmtestr` defaults to using `ChatLLM7`.\n- `api_key` (*Optional[str]*): Your API key for `LLM7`. If not provided, it will attempt to fetch from the environment variable `LLM7_API_KEY`.\n\n## Underlying LLM\n\nBy default, `llmtestr` uses the `ChatLLM7` class from `langchain_llm7`, which you can find on PyPI: [langchain_llm7](https://pypi.org/project/langchain-llm7/). You can also pass your custom LLM instance based on the `langchain` interface for different providers like OpenAI, Anthropic, Google, etc.\n\n### Examples:\n\n**Using OpenAI:**\n\n```python\nfrom langchain_openai import ChatOpenAI\nfrom llmtestr import llmtestr\n\nllm = ChatOpenAI()\nresponse = llmtestr(user_input=\"Test prompt\", llm=llm)\n```\n\n**Using Anthropic:**\n\n```python\nfrom langchain_anthropic import ChatAnthropic\nfrom llmtestr import llmtestr\n\nllm = ChatAnthropic()\nresponse = llmtestr(user_input=\"Test prompt\", llm=llm)\n```\n\n**Using Google Generative AI:**\n\n```python\nfrom langchain_google_genai import ChatGoogleGenerativeAI\nfrom llmtestr import llmtestr\n\nllm = ChatGoogleGenerativeAI()\nresponse = llmtestr(user_input=\"Test prompt\", llm=llm)\n```\n\n## Rate Limits \u0026 API Keys\n\nThe default free tier for LLM7 provides sufficient rate limits for most development needs. To get higher limits, you can:\n\n- Set the environment variable `LLM7_API_KEY`.\n- Pass your API key directly:\n\n```python\nresponse = llmtestr(user_input=\"Test prompt\", api_key=\"your_api_key\")\n```\n\nRegister for a free API key at [https://token.llm7.io/](https://token.llm7.io/).\n\n## Support \u0026 Issues\n\nFor issues, bugs, or feature requests, please visit the GitHub repo issues page: [https://github.com/chigwell/llmtestr/issues](https://github.com/chigwell/llmtestr/issues).\n\n## Author\n\nEugene Evstafev  \nEmail: hi@euegne.plus  \nGitHub: [chigwell](https://github.com/chigwell)","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchigwell%2Fllmtestr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchigwell%2Fllmtestr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchigwell%2Fllmtestr/lists"}