{"id":26192734,"url":"https://github.com/antiwork/shortest","last_synced_at":"2025-05-12T15:34:45.838Z","repository":{"id":259780570,"uuid":"859537060","full_name":"antiwork/shortest","owner":"antiwork","description":"QA via natural language AI tests","archived":false,"fork":false,"pushed_at":"2025-04-03T23:28:23.000Z","size":2074,"stargazers_count":4884,"open_issues_count":0,"forks_count":278,"subscribers_count":27,"default_branch":"main","last_synced_at":"2025-05-12T01:11:18.883Z","etag":null,"topics":["anthropic","automation","chromium","e2e-testing","e2e-tests","end-to-end-testing","javascript","nextjs","playwright","test-automation","testing","testing-framework","testing-tool"],"latest_commit_sha":null,"homepage":"https://shortest.com","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/antiwork.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"license.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-09-18T20:44:05.000Z","updated_at":"2025-05-11T22:02:31.000Z","dependencies_parsed_at":"2024-12-07T06:24:35.465Z","dependency_job_id":"448dd435-6026-45df-aecd-280a13734910","html_url":"https://github.com/antiwork/shortest","commit_stats":null,"previous_names":["anti-work/shortest","antiwork/shortest"],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antiwork%2Fshortest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antiwork%2Fshortest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antiwork%2Fshortest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/antiwork%2Fshortest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/antiwork","download_url":"https://codeload.github.com/antiwork/shortest/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253656410,"owners_count":21943081,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anthropic","automation","chromium","e2e-testing","e2e-tests","end-to-end-testing","javascript","nextjs","playwright","test-automation","testing","testing-framework","testing-tool"],"created_at":"2025-03-12T01:24:31.263Z","updated_at":"2025-05-12T15:34:45.807Z","avatar_url":"https://github.com/antiwork.png","language":"TypeScript","funding_links":[],"categories":["TypeScript","\u003ca name=\"TypeScript\"\u003e\u003c/a\u003eTypeScript"],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/user-attachments/assets/57d23950-206b-4640-a649-66a175660ade\" alt=\"Shortest logo\" width=\"128\" /\u003e\n\u003c/p\u003e\n\n# Shortest\n\nAI-powered natural language end-to-end testing framework.\n\n\u003cvideo src=\"https://github.com/user-attachments/assets/d443279e-7364-452b-9f50-0c8dd0cf55fc\" controls autoplay loop muted\u003e\nYour browser does not support the video tag.\n\u003c/video\u003e\n\n## Features\n\n- Natural language E2E testing framework\n- AI-powered test execution using Anthropic Claude API\n- Built on Playwright\n- GitHub integration with 2FA support\n- Email validation with Mailosaur\n\n## Using Shortest in your project\n\nIf helpful, [here's a short video](https://github.com/antiwork/shortest/issues/143#issuecomment-2564488173)!\n\n### Installation\n\nUse the `shortest init` command to streamline the setup process in a new or existing project.\n\nThe `shortest init` command will:\n\n```sh\nnpx @antiwork/shortest init\n```\n\nThis will:\n\n- Automatically install the `@antiwork/shortest` package as a dev dependency if it is not already installed\n- Create a default `shortest.config.ts` file with boilerplate configuration\n- Generate a `.env.local` file (unless present) with placeholders for required environment variables, such as `ANTHROPIC_API_KEY`\n- Add `.env.local` and `.shortest/` to `.gitignore`\n\n### Quick start\n\n1. Determine your test entry and add your Anthropic API key in config file: `shortest.config.ts`\n\n```typescript\nimport type { ShortestConfig } from \"@antiwork/shortest\";\n\nexport default {\n  headless: false,\n  baseUrl: \"http://localhost:3000\",\n  browser: {\n    contextOptions: {\n      ignoreHTTPSErrors: true\n    },\n  },\n  testPattern: \"**/*.test.ts\",\n  ai: {\n    provider: \"anthropic\",\n  },\n} satisfies ShortestConfig;\n```\nThe Anthropic API key defaults to `SHORTEST_ANTHROPIC_API_KEY` / `ANTHROPIC_API_KEY` environment variables. Can be overwritten via `ai.config.apiKey`.\n\nOptionally, you can configure browser behavior using the `browser.contextOptions` property in your configuration file. This allows you to pass custom [Playwright browser context options](https://playwright.dev/docs/api/class-browser#browser-new-context).\n\n2. Create test files using the pattern specified in the config: `app/login.test.ts`\n\n```typescript\nimport { shortest } from \"@antiwork/shortest\";\n\nshortest(\"Login to the app using email and password\", {\n  username: process.env.GITHUB_USERNAME,\n  password: process.env.GITHUB_PASSWORD,\n});\n```\n\n### Using callback functions\n\nYou can also use callback functions to add additional assertions and other logic. AI will execute the callback function after the test\nexecution in browser is completed.\n\n```typescript\nimport { shortest } from \"@antiwork/shortest\";\nimport { db } from \"@/lib/db/drizzle\";\nimport { users } from \"@/lib/db/schema\";\nimport { eq } from \"drizzle-orm\";\n\nshortest(\"Login to the app using username and password\", {\n  username: process.env.USERNAME,\n  password: process.env.PASSWORD,\n}).after(async ({ page }) =\u003e {\n  // Get current user's clerk ID from the page\n  const clerkId = await page.evaluate(() =\u003e {\n    return window.localStorage.getItem(\"clerk-user\");\n  });\n\n  if (!clerkId) {\n    throw new Error(\"User not found in database\");\n  }\n\n  // Query the database\n  const [user] = await db\n    .select()\n    .from(users)\n    .where(eq(users.clerkId, clerkId))\n    .limit(1);\n\n  expect(user).toBeDefined();\n});\n```\n\n### Lifecycle hooks\n\nYou can use lifecycle hooks to run code before and after the test.\n\n```typescript\nimport { shortest } from \"@antiwork/shortest\";\n\nshortest.beforeAll(async ({ page }) =\u003e {\n  await clerkSetup({\n    frontendApiUrl:\n      process.env.PLAYWRIGHT_TEST_BASE_URL ?? \"http://localhost:3000\",\n  });\n});\n\nshortest.beforeEach(async ({ page }) =\u003e {\n  await clerk.signIn({\n    page,\n    signInParams: {\n      strategy: \"email_code\",\n      identifier: \"iffy+clerk_test@example.com\",\n    },\n  });\n});\n\nshortest.afterEach(async ({ page }) =\u003e {\n  await page.close();\n});\n\nshortest.afterAll(async ({ page }) =\u003e {\n  await clerk.signOut({ page });\n});\n```\n\n### Chaining tests\n\nShortest supports flexible test chaining patterns:\n\n```typescript\n// Sequential test chain\nshortest([\n  \"user can login with email and password\",\n  \"user can modify their account-level refund policy\",\n]);\n\n// Reusable test flows\nconst loginAsLawyer = \"login as lawyer with valid credentials\";\nconst loginAsContractor = \"login as contractor with valid credentials\";\nconst allAppActions = [\"send invoice to company\", \"view invoices\"];\n\n// Combine flows with spread operator\nshortest([loginAsLawyer, ...allAppActions]);\nshortest([loginAsContractor, ...allAppActions]);\n```\n\n### API testing\n\nTest API endpoints using natural language\n\n```typescript\nconst req = new APIRequest({\n  baseURL: API_BASE_URI,\n});\n\nshortest(\n  \"Ensure the response contains only active users\",\n  req.fetch({\n    url: \"/users\",\n    method: \"GET\",\n    params: new URLSearchParams({\n      active: true,\n    }),\n  }),\n);\n```\n\nOr simply:\n\n```typescript\nshortest(`\n  Test the API GET endpoint ${API_BASE_URI}/users with query parameter { \"active\": true }\n  Expect the response to contain only active users\n`);\n```\n\n### Running tests\n\n```bash\npnpm shortest                   # Run all tests\npnpm shortest login.test.ts     # Run specific tests from a file\npnpm shortest login.test.ts:23  # Run specific test from a file using a line number\npnpm shortest --headless        # Run in headless mode using\n```\n\nYou can find example tests in the [`examples`](./examples) directory.\n\n### CI setup\n\nYou can run Shortest in your CI/CD pipeline by running tests in headless mode. Make sure to add your Anthropic API key to your CI/CD pipeline secrets.\n\n[See example here](https://github.com/antiwork/shortest/blob/main/.github/workflows/shortest.yml)\n\n### GitHub 2FA login setup\n\nShortest supports login using GitHub 2FA. For GitHub authentication tests:\n\n1. Go to your repository settings\n2. Navigate to \"Password and Authentication\"\n3. Click on \"Authenticator App\"\n4. Select \"Use your authenticator app\"\n5. Click \"Setup key\" to obtain the OTP secret\n6. Add the OTP secret to your `.env.local` file or use the Shortest CLI to add it\n7. Enter the 2FA code displayed in your terminal into Github's Authenticator setup page to complete the process\n\n```bash\nshortest --github-code --secret=\u003cOTP_SECRET\u003e\n```\n\n### Environment setup\n\nRequired in `.env.local`:\n\n```bash\nANTHROPIC_API_KEY=your_api_key\nGITHUB_TOTP_SECRET=your_secret  # Only for GitHub auth tests\n```\n\n## Shortest CLI development\n\nThe [NPM package](https://www.npmjs.com/package/@antiwork/shortest) is located in [`packages/shortest/`](./packages/shortest). See [CONTRIBUTING](./packages/shortest/CONTRIBUTING.md) guide.\n\n## Web app development\n\nThis guide will help you set up the Shortest web app for local development.\n\n### Prerequisites\n\n- React \u003e=19.0.0 (if using with Next.js 14+ or Server Actions)\n- Next.js \u003e=14.0.0 (if using Server Components/Actions)\n\n\u003e [!WARNING]\n\u003e Using this package with React 18 in Next.js 14+ projects may cause type conflicts with Server Actions and `useFormStatus`\n\u003e\n\u003e If you encounter type errors with form actions or React hooks, ensure you're using React 19\n\n### Getting started\n\n1. Clone the repository:\n\n```bash\ngit clone https://github.com/antiwork/shortest.git\ncd shortest\n```\n\n2. Install dependencies:\n\n```bash\nnpm install -g pnpm\npnpm install\n```\n\n### Environment setup\n\n#### For Antiwork team members\n\nPull Vercel env vars:\n\n```bash\npnpm i -g vercel\nvercel link\nvercel env pull\n```\n\n#### For other contributors\n\n1. Run `pnpm run setup` to configure the environment variables.\n2. The setup wizard will ask you for information. Refer to \"Services Configuration\" section below for more details.\n\n### Set up the database\n\n```bash\npnpm drizzle-kit generate\npnpm db:migrate\npnpm db:seed # creates stripe products, currently unused\n```\n\n### Services configuration\n\nYou'll need to set up the following services for local development. If you're not an Antiwork Vercel team member, you'll need to either run the setup wizard `pnpm run setup` or manually configure each of these services and add the corresponding environment variables to your `.env.local` file:\n\n\u003cdetails\u003e\n\u003csummary\u003eClerk\u003c/summary\u003e\n\n1. Go to [clerk.com](https://clerk.com) and create a new app.\n2. Name it whatever you like and **disable all login methods except GitHub**.\n   ![Clerk App Login](https://github.com/user-attachments/assets/1de7aebc-8e9d-431a-ae13-af60635307a1)\n3. Once created, copy the environment variables to your `.env.local` file.\n   ![Clerk Env Variables](https://github.com/user-attachments/assets/df3381e6-017a-4e01-8bd3-5793e5f5d31e)\n4. In the Clerk dashboard, disable the \"Require the same device and browser\" setting to ensure tests with Mailosaur work properly.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eVercel Postgres\u003c/summary\u003e\n\n1. Go to your dashboard at [vercel.com](https://vercel.com).\n2. Navigate to the Storage tab and click the `Create Database` button.\n   ![Vercel Create Database](https://github.com/user-attachments/assets/acdf3ba7-31a6-498b-860c-171018d5ba02)\n3. Choose `Postgres` from the `Browse Storage` menu.\n   ![Neon Postgres](https://github.com/user-attachments/assets/9ad2a391-5213-4f31-a6c3-b9e54c69bb2e)\n4. Copy your environment variables from the `Quickstart` `.env.local` tab.\n   ![Vercel Postgres .env.local](https://github.com/user-attachments/assets/e48f1d96-2fd6-4e2e-aaa6-eeb5922cc521)\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eAnthropic\u003c/summary\u003e\n\n1. Go to your dashboard at [anthropic.com](https://anthropic.com) and grab your API Key.\n   - Note: If you've never done this before, you will need to answer some questions and likely load your account with a balance. Not much is needed to test the app.\n     ![Anthropic API Key](https://github.com/user-attachments/assets/0905ed4b-5815-4d50-bf43-8713a4397674)\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eStripe\u003c/summary\u003e\n\n1. Go to your `Developers` dashboard at [stripe.com](https://stripe.com).\n2. Turn on `Test mode`.\n3. Go to the `API Keys` tab and copy your `Secret key`.\n   ![Stripe Secret Key](https://github.com/user-attachments/assets/0830b226-f2c2-4b92-a28f-f4682ad03ec0)\n4. Go to the terminal of your project and type `pnpm run stripe:webhooks`. It will prompt you to login with a code then give you your `STRIPE_WEBHOOK_SECRET`.\n   ![Stripe Webhook Secret](https://github.com/user-attachments/assets/b02531ed-5c31-40ba-8483-32880aa3ca36)\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eGitHub OAuth\u003c/summary\u003e\n\n1. Create a GitHub OAuth App:\n\n   - Go to your GitHub account settings.\n   - Navigate to `Developer settings` \u003e `OAuth Apps` \u003e `New OAuth App`.\n   - Fill in the application details:\n     - **Application name**: Choose any name for your app\n     - **Homepage URL**: Set to `http://localhost:3000` for local development\n     - **Authorization callback URL**: Use the Clerk-provided callback URL (found in below image)\n       ![Github OAuth App](https://github.com/user-attachments/assets/1af635fd-dedc-401c-a45a-159cb20bb209)\n\n2. Configure Clerk with GitHub OAuth:\n   - Go to your Clerk dashboard.\n   - Navigate to `Configure` \u003e `SSO Connections` \u003e `GitHub`.\n   - Select `Use custom credentials`\n   - Enter your `Client ID` and `Client Secret` from the GitHub OAuth app you just created.\n   - Add `repo` to the `Scopes`\n     ![Clerk Custom Credentials](https://github.com/user-attachments/assets/31d414e1-4e1e-4725-8649-ec1826c6e53e)\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eMailosaur\u003c/summary\u003e\n\n1. [Sign up](https://mailosaur.com/app/signup) for an account with Mailosaur.\n2. Create a new Inbox/Server.\n3. Go to [API Keys](https://mailosaur.com/app/keys) and create a standard key.\n4. Update the environment variables:\n   - `MAILOSAUR_API_KEY`: Your API key\n   - `MAILOSAUR_SERVER_ID`: Your server ID\n\nThe email used to test the login flow will have the format `shortest@\u003cMAILOSAUR_SERVER_ID\u003e.mailosaur.net`, where\n`MAILOSAUR_SERVER_ID` is your server ID.\nMake sure to add the email as a new user under the Clerk app.\n\n\u003c/details\u003e\n\n### Running locally\n\nRun the development server:\n\n```bash\npnpm dev\n```\n\nOpen [http://localhost:3000](http://localhost:3000) in your browser to see the app in action.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fantiwork%2Fshortest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fantiwork%2Fshortest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fantiwork%2Fshortest/lists"}