{"id":33314333,"url":"https://github.com/dawiddiwad/checkmate","last_synced_at":"2026-05-30T22:01:31.408Z","repository":{"id":324615288,"uuid":"1097759805","full_name":"dawiddiwad/checkmate","owner":"dawiddiwad","description":"AI-Driven Test Automation Framework powered by OpenAI API \u0026 Playwright","archived":false,"fork":false,"pushed_at":"2026-05-23T18:55:33.000Z","size":4251,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-05-23T20:25:54.366Z","etag":null,"topics":["claude","gemini","grok","groq","low-code","openai","playwright","salesforce","test-automation"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dawiddiwad.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"docs/ROADMAP.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2025-11-16T19:19:36.000Z","updated_at":"2026-05-23T18:53:48.000Z","dependencies_parsed_at":"2026-03-01T10:15:41.780Z","dependency_job_id":"7547287a-78b2-46ab-a2a5-cd6300bcdb69","html_url":"https://github.com/dawiddiwad/checkmate","commit_stats":null,"previous_names":["dawiddiwad/checkmate"],"tags_count":18,"template":false,"template_full_name":null,"purl":"pkg:github/dawiddiwad/checkmate","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dawiddiwad%2Fcheckmate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dawiddiwad%2Fcheckmate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dawiddiwad%2Fcheckmate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dawiddiwad%2Fcheckmate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dawiddiwad","download_url":"https://codeload.github.com/dawiddiwad/checkmate/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dawiddiwad%2Fcheckmate/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33711018,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-30T02:00:06.278Z","response_time":92,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["claude","gemini","grok","groq","low-code","openai","playwright","salesforce","test-automation"],"created_at":"2025-11-19T12:00:35.477Z","updated_at":"2026-05-30T22:01:31.368Z","avatar_url":"https://github.com/dawiddiwad.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# **_checkmate_**\n\nAI test automation that actually works. Write tests in plain English, without locators, and with less code.\n\n![playwright](https://img.shields.io/badge/Playwright-1.60.0-blue.svg)\n![typescript](https://img.shields.io/badge/TypeScript-5.9.3-blue.svg)\n![nodejs](https://img.shields.io/badge/Node.js-LTS-green.svg)\n![openai](https://img.shields.io/badge/OpenAI-API-yellow.svg)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n\n##\n\n```typescript\nawait ai.run({\n\taction: `\n\t\tNavigate to google.com\n\t\tType 'playwright test automation' in the search bar\n\t\tPress Enter key`,\n\texpect: `\n\t\tSearch results contain the playwright.dev link`,\n})\n```\n\n##\n\n✅ **Zero Locators** - Write tests in plain English  \n✅ **Any Provider** - Gemini, Claude, Groq, GPT, xAI, or local models  \n✅ **Web \u0026 Salesforce** - Basic support out of the box  \n✅ **Cost Optimized** - Built-in token management and budgeting  \n✅ **Playwright Test** - Native reports, traces and debugging  \n✅ **Fully Customizable** - Build your own [extensions](docs/EXTENSIONS.md) and tools\n\n\u003cimg src=\"docs/img/gpt-oss-20b-e2e-checkout.gif\" alt=\"example-e2e-test\" width=\"100%\"/\u003e\n\n## Get Started in 5 Minutes\n\n### Prerequisites\n\n- Node.js [LTS](https://nodejs.org/en/download)\n- OpenAI [API key](https://platform.openai.com/api-keys) or compatible provider [Groq](https://console.groq.com/keys) [Gemini](https://aistudio.google.com/app/api-keys) [xAI](https://x.ai/api) etc.\n\n### 1. Install\n\n```bash\nnpm install -D dotenv @playwright/test @xoxoai/checkmate\nnpx playwright install\n```\n\n### 2. Configure `.env`\n\n_using [OpenAI API](https://platform.openai.com/settings/organization/api-keys) key and default settings:_\n\n```bash\nOPENAI_API_KEY=#your_api_key_here\n```\n\n_for other providers, set the base url and model:_\n\n```bash\nOPENAI_BASE_URL=https://api.groq.com/openai/v1\nOPENAI_MODEL=openai/gpt-oss-20b\n```\n\n### 3. Scaffold Test Examples\n\n```bash\nnpx checkmate create-examples\n```\n\n### 4. Run Tests\n\n```bash\nnpm run test:web:example\n```\n\n### 5. View Report\n\n```bash\nnpm run show:report\n```\n\n## Writing Tests\n\n**_checkmate_** tests are written using natural language by specifying `action` and `expect`:\n\n```typescript\nimport { test } from '@xoxoai/checkmate/playwright'\n\ntest.describe('multi-step : full AI mode', async () =\u003e {\n\ttest('purchase flow', async ({ ai }) =\u003e {\n\t\tawait test.step('Open Shop', async () =\u003e {\n\t\t\tawait ai.run({\n\t\t\t\taction: `\n\t\t\t\tNavigate to https://my-shop.com`,\n\t\t\t\texpect: `\n\t\t\t\tMy Shop home page is loaded`,\n\t\t\t})\n\t\t})\n\n\t\tawait test.step('Select product', async () =\u003e {\n\t\t\tawait ai.run({\n\t\t\t\taction: `\n\t\t\t\tClick 'Shop Now' on 'Men's Outerwear' category\n\t\t\t\tClick on the first Shell product in the list`,\n\t\t\t\texpect: `\n\t\t\t\tProduct detail with title and price.`,\n\t\t\t})\n\t\t})\n\n\t\tawait test.step('Cart and checkout', async () =\u003e {\n\t\t\tawait ai.run({\n\t\t\t\taction: `\n\t\t\t\tClick 'Add to Cart'\n\t\t\t\tClick 'Checkout' in the 'Added to cart' dialog`,\n\t\t\t\texpect: `\n\t\t\t\tCheckout with Order Summary and totals`,\n\t\t\t})\n\t\t})\n\t})\n})\n```\n\nThat's it. No page objects, no selectors. No locators. Peace on Earth.\n\nTests are orchestrated by [playwright](https://playwright.dev/docs/test-configuration) [config](playwright.config.ts).\n\n### API\n\nCompose your own **_checkmate_** using [extensions](docs/EXTENSIONS.md):\n\n```typescript\nimport { createRunner } from '@xoxoai/checkmate/core'\nimport { web } from '@xoxoai/checkmate/playwright'\nimport { notion, database, api } from 'my-custom-extensions'\n\nconst ai = createRunner({\n\textensions: [web({ page }), notion(), database(), api()],\n})\n\nawait ai.run({\n\taction: 'Open the pricing page',\n\texpect: 'Pricing details are visible',\n})\n```\n\n### Entry Points:\n\n`@xoxoai/checkmate/core`: compose runner, tools, and extensions.  \n`@xoxoai/checkmate/playwright`: Web extension with Playwright `test` and `expect`.  \n`@xoxoai/checkmate/salesforce`: Salesforce extensions with the same `ai` fixture shape.\n\nSee [guide](docs/GUIDE.md#best-practices) for tips on writing effective tests.\n\n## Costs\n\nThey depend on the model, provider, test complexity, and number of steps.\n\nEstimates for [gpt-oss-20b hosted on groq.com](https://console.groq.com/docs/model/openai/gpt-oss-20b):\n\n- Simple test (~5 steps): ~$0.001 - $0.01\n- Complex test (~20 steps): ~$0.01 - $0.05\n- Full E2E suite (~50 complex tests): ~$1.00 - $2.00\n\n**_checkmate_** includes built-in token usage [monitoring](docs/GUIDE.md#cost-management).\n\nSee [guide](docs/GUIDE.md#cost-management) for cost control and monitoring options.\n\n## Common Issues\n\n**AI makes incorrect decisions**\n\n- Provide precise descriptions in `action` and focused assertions in `expect`\n- Reference specific element and roles, for example: text, label, button, list, etc.\n- Break complex workflows into single-action steps and use a step-by-step approach\n\n**Tests loop during step execution**\n\n- Increase `OPENAI_TEMPERATURE` to encourage exploration\n- Use a reasoning model if possible to improve accuracy\n\n**High token costs**\n\n- Enable [snapshot filtering](docs/GUIDE.md#using-snapshot-filtering-for-token-optimization) with `CHECKMATE_SNAPSHOT_FILTERING=true` auto-filter elements\n- Adjust reasoning effort: `OPENAI_REASONING_EFFORT`\n- Consider disabling `OPENAI_INCLUDE_SCREENSHOT_IN_SNAPSHOT` if visuals are not needed\n- Use a cheaper model, lower-end models often perform well: `gpt-5.4-nano` or `gpt-oss-20b`\n\nSee [guide](docs/GUIDE.md#openai-api-settings) for detailed configuration options and tips.\n\n## FAQ\n\n**Which models work best?**  \nYou can use any model that was trained for tool use.\n\nHere are the best picks based on extensive testing:\n\n- Highly recommended: [`gpt-oss-20b` hosted on groq.com](https://console.groq.com/docs/model/openai/gpt-oss-20b). Groq's infrastructure is optimized for minimal latency and fast inference, making it ideal for E2E test automation.\n- Google's `gemini-2.5-flash` offers an excellent balance of cost and performance if you prefer major cloud providers.\n- OpenAI's `gpt-5-mini`, `gpt-5.4-nano` and xAI's `grok-4-1-fast-reasoning` also work well and keep costs relatively low.\n\n**Can I use local models?**  \nYes - **_checkmate_** works with any OpenAI‑compatible API, including local models via LM Studio, Ollama, or llama.cpp. I recommend [qwen3.5-4b](https://huggingface.co/Qwen/Qwen3.5-4B). It is fast (≈100 tokens/sec on an RTX 3060 Ti; ≈40 tokens/sec on Apple M3) and performs surprisingly well for E2E testing.\n\n**Does it work with CI/CD?**  \nAbsolutely. Use **_checkmate_** as part of your existing [Playwright Test suites in any CI/CD pipeline](https://playwright.dev/docs/best-practices#run-tests-on-ci). You can mix AI‑driven steps and traditional tests as needed.\n\n**Is this production-ready?**  \nIt depends. If you can accept some non‑deterministic behavior and leverage LLMs' randomness to help address the [pesticide paradox](https://medium.com/@suwekasansiluni/the-pesticide-paradox-what-farming-teaches-us-about-software-testing-ab5d625d4de1), **_checkmate_** can be production-ready. In many cases, the maintenance savings, faster development, and benefits of non‑linear execution outweigh occasional hiccups.\n\nIf you require 100% deterministic tests at all times, traditional Playwright remains the better choice.\n\n**Best part?**  \nYou can mix both approaches within the same test suite, combining AI‑driven and traditional tests as needed:\n\n```typescript\n// traditional playwright actions:\nawait page.goto('https://www.google.com')\nconst searchBox = page.getByRole('combobox', { name: 'Search', exact: true })\nawait searchBox.fill('playwright test automation')\nawait searchBox.press('Enter')\n\n// ai-driven actions and assertions:\nawait ai.run({\n\taction: 'Click on the link that leads to playwright.dev',\n\texpect: 'The playwright.dev homepage is displayed',\n})\n```\n\n## Documentation\n\n- [**_checkmate_** guide](docs/GUIDE.md)\n- [**_checkmate_** extensions](docs/EXTENSIONS.md)\n- [**playwright** official website](https://playwright.dev/)\n\n## Contributing\n\nI'd love your help! Key areas:\n\n- Additional tool integrations (API testing, Salesforce, etc.)\n- Further cost optimization techniques\n- Context and prompt engineering improvements\n- Error handling and recovery\n\nSee [roadmap](docs/ROADMAP.md) for future plans and development\n\n## License\n\nMIT [license](LICENSE)\n\n## Why I build this?\n\nTest automation shouldn't require a PhD in XPath. This project explores how AI can make it accessible to anyone.\n\nLess coding, more testing.\n\nBuilt with ❤️ by [Dawid Dobrowolski](https://github.com/dawiddiwad)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdawiddiwad%2Fcheckmate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdawiddiwad%2Fcheckmate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdawiddiwad%2Fcheckmate/lists"}