Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/anti-work/shortest
QA via natural language AI tests
https://github.com/anti-work/shortest
Last synced: 6 days ago
JSON representation
QA via natural language AI tests
- Host: GitHub
- URL: https://github.com/anti-work/shortest
- Owner: anti-work
- License: mit
- Created: 2024-09-18T20:44:05.000Z (4 months ago)
- Default Branch: main
- Last Pushed: 2024-12-30T21:47:09.000Z (6 days ago)
- Last Synced: 2024-12-30T22:27:32.784Z (6 days ago)
- Language: TypeScript
- Homepage: https://shortest.com
- Size: 1.28 MB
- Stars: 3,355
- Watchers: 24
- Forks: 178
- Open Issues: 14
-
Metadata Files:
- Readme: README.md
- License: license.md
Awesome Lists containing this project
- awesome-AI-driven-development - shortest - QA via natural language AI tests (Uncategorized / Uncategorized)
README
# Shortest
AI-powered natural language end-to-end testing framework.
Your browser does not support the video tag.
## Features
- Natural language E2E testing framework
- AI-powered test execution using Anthropic Claude API
- Built on Playwright
- GitHub integration with 2FA support
- Email validation with Mailosaur
## Using Shortest in your project
If helpful, [here's a short video](https://github.com/anti-work/shortest/issues/143#issuecomment-2564488173)!
### Installation
```bash
npm install -D @antiwork/shortest
# or
pnpm add -D @antiwork/shortest
# or
yarn add -D @antiwork/shortest
```
Add `.shortest/` to your `.gitignore` (where Shortest stores screenshots of each test run):
```bash
echo ".shortest/" >> .gitignore
```
If you installed shortest without `-g` flag, you can run tests as follows:
```bash
npx shortest # for npm
pnpm shortest # for pnpm
yarn shortest # for yarn
```
### Quick start
1. Determine your test entry and add your Anthropic API key in config file: `shortest.config.ts`
```typescript
import type { ShortestConfig } from '@antiwork/shortest';
export default {
headless: false,
baseUrl: 'http://localhost:3000',
testDir: 'app/__tests__',
anthropicKey: process.env.ANTHROPIC_API_KEY
} satisfies ShortestConfig;
```
2. Write your test in your test directory: `app/__tests__/login.test.ts`
```typescript
import { shortest } from '@antiwork/shortest'
shortest('Login to the app using email and password', { username: process.env.GITHUB_USERNAME, password: process.env.GITHUB_PASSWORD })
```
### Using callback functions
You can also use callback functions to add additional assertions and other logic. AI will execute the callback function after the test
execution in browser is completed.
```typescript
import { shortest } from '@antiwork/shortest';
import { db } from '@/lib/db/drizzle';
import { users } from '@/lib/db/schema';
import { eq } from 'drizzle-orm';
shortest('Login to the app using username and password', {
username: process.env.USERNAME,
password: process.env.PASSWORD
}).after(async ({ page }) => {
// Get current user's clerk ID from the page
const clerkId = await page.evaluate(() => {
return window.localStorage.getItem('clerk-user');
});
if (!clerkId) {
throw new Error('User not found in database');
}
// Query the database
const [user] = await db
.select()
.from(users)
.where(eq(users.clerkId, clerkId))
.limit(1);
expect(user).toBeDefined();
});
```
### Lifecycle hooks
You can use lifecycle hooks to run code before and after the test.
```typescript
import { shortest } from '@antiwork/shortest';
shortest.beforeAll(async ({ page }) => {
await clerkSetup({
frontendApiUrl: process.env.PLAYWRIGHT_TEST_BASE_URL ?? "http://localhost:3000",
});
});
shortest.beforeEach(async ({ page }) => {
await clerk.signIn({
page,
signInParams: {
strategy: "email_code",
identifier: "[email protected]"
},
});
});
shortest.afterEach(async ({ page }) => {
await page.close();
});
shortest.afterAll(async ({ page }) => {
await clerk.signOut({ page });
});
```
### Running tests
```bash
pnpm shortest # Run all tests
pnpm shortest login.test.ts # Run specific test
pnpm shortest --headless # Run in headless mode using cli
```
You can find example tests in the [`examples`](./examples) directory.
### GitHub 2FA login setup
Shortest currently supports login using Github 2FA. For GitHub authentication tests:
1. Go to your repository settings
2. Navigate to "Password and Authentication"
3. Click on "Authenticator App"
4. Select "Use your authenticator app"
5. Click "Setup key" to obtain the OTP secret
6. Add the OTP secret to your `.env.local` file or use the Shortest CLI to add it
7. Enter the 2FA code displayed in your terminal into Github's Authenticator setup page to complete the process
```bash
shortest --github-code --secret=
```
### Environment setup
Required in `.env.local`:
```bash
ANTHROPIC_API_KEY=your_api_key
GITHUB_TOTP_SECRET=your_secret # Only for GitHub auth tests
```
### CI setup
You can run Shortest in your CI/CD pipeline by running tests in headless mode. Make sure to add your Anthropic API key to your CI/CD pipeline secrets.
## Web app development
This guide will help you set up the Shortest web app for local development.
### Prerequisites
- React >=19.0.0 (if using with Next.js 14+ or Server Actions)
- Next.js >=14.0.0 (if using Server Components/Actions)
> [!WARNING]
> Using this package with React 18 in Next.js 14+ projects may cause type conflicts with Server Actions and `useFormStatus`
>
> If you encounter type errors with form actions or React hooks, ensure you're using React 19
### Getting started
1. Clone the repository:
```bash
git clone https://github.com/anti-work/shortest.git
cd shortest
```
2. Install dependencies:
```bash
npm install -g pnpm
pnpm install
```
### Environment setup
#### For Anti-Work team members
Pull Vercel env vars:
```bash
pnpm i -g vercel
vercel link
vercel env pull
```
#### For other contributors
1. Run `pnpm run setup` to configure the environment variables.
2. The setup wizard will ask you for information. Refer to "Services Configuration" section below for more details.
### Set up the database
```bash
pnpm drizzle-kit generate
pnpm db:migrate
pnpm db:seed # creates stripe products, currently unused
```
### Services configuration
You'll need to set up the following services for local development. If you're not a Anti-Work Vercel team member, you'll need to either run the setup wizard `pnpm run setup` or manually configure each of these services and add the corresponding environment variables to your `.env.local` file:
Clerk
1. Go to [clerk.com](https://clerk.com) and create a new app.
2. Name it whatever you like and **disable all login methods except GitHub**.
3. Once created, copy the environment variables to your `.env.local` file.
4. In the Clerk dashboard, disable the "Require the same device and browser" setting to ensure tests with Mailosaur work properly.
Vercel Postgres
1. Go to your dashboard at [vercel.com](https://vercel.com).
2. Navigate to the Storage tab and click the `Create Database` button.
3. Choose `Postgres` from the `Browse Storage` menu.
4. Copy your environment variables from the `Quickstart` `.env.local` tab.
Anthropic
1. Go to your dashboard at [anthropic.com](https://anthropic.com) and grab your API Key.
- Note: If you've never done this before, you will need to answer some questions and likely load your account with a balance. Not much is needed to test the app.
Stripe
1. Go to your `Developers` dashboard at [stripe.com](https://stripe.com).
2. Turn on `Test mode`.
3. Go to the `API Keys` tab and copy your `Secret key`.
4. Go to the terminal of your project and type `pnpm run stripe:webhooks`. It will prompt you to login with a code then give you your `STRIPE_WEBHOOK_SECRET`.
GitHub OAuth
1. Create a GitHub OAuth App:
- Go to your GitHub account settings.
- Navigate to `Developer settings` > `OAuth Apps` > `New OAuth App`.
- Fill in the application details:
- **Application name**: Choose any name for your app
- **Homepage URL**: Set to `http://localhost:3000` for local development
- **Authorization callback URL**: Use the Clerk-provided callback URL (found in below image)
2. Configure Clerk with GitHub OAuth:
- Go to your Clerk dashboard.
- Navigate to `Configure` > `SSO Connections` > `GitHub`.
- Select `Use custom credentials`
- Enter your `Client ID` and `Client Secret` from the GitHub OAuth app you just created.
- Add `repo` to the `Scopes`
Mailosaur
1. Go to [mailosaur.com](https://mailosaur.com) and create an account.
2. Create a new server and copy the Server ID.
3. Go to your API settings and copy your API key.
- You'll need both the Server ID and API key for your environment variables:
- `MAILOSAUR_API_KEY`: Your API key
- `MAILOSAUR_SERVER_ID`: Your server ID
### Running locally
Run the development server:
```bash
pnpm dev
```
Open [http://localhost:3000](http://localhost:3000) in your browser to see the app in action.
## Shortest CLI development
1. Make changes to the package source code in `packages/shortest/`
2. Test changes instantly during development (no build needed):
```bash
pnpm shortest:dev -h
```
3. To test the actual built package:
```bash
pnpm build:pkg
pnpm shortest --help
```
4. To test in another project:
```bash
# In Shortest package directory
cd packages/shortest
pnpm pack
# In your test project
npm install /path/to/antiwork-shortest-{version}.tgz.tgz
npx shortest -h
```