{"id":51528754,"url":"https://github.com/browserstack/ai-sdk-sample-app","last_synced_at":"2026-07-09T00:31:50.100Z","repository":{"id":360383520,"uuid":"1232898240","full_name":"browserstack/ai-sdk-sample-app","owner":"browserstack","description":"This repository provides a comprehensive sample application designed to demonstrate the core functionality and integration patterns of the AI Evals SDK.","archived":false,"fork":false,"pushed_at":"2026-07-08T09:49:27.000Z","size":175,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-08T11:14:10.712Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/browserstack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-08T11:42:05.000Z","updated_at":"2026-07-08T09:47:20.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/browserstack/ai-sdk-sample-app","commit_stats":null,"previous_names":["browserstack/ai-sdk-sample-app"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/browserstack/ai-sdk-sample-app","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/browserstack%2Fai-sdk-sample-app","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/browserstack%2Fai-sdk-sample-app/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/browserstack%2Fai-sdk-sample-app/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/browserstack%2Fai-sdk-sample-app/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/browserstack","download_url":"https://codeload.github.com/browserstack/ai-sdk-sample-app/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/browserstack%2Fai-sdk-sample-app/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35282897,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-08T02:00:06.796Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-09T00:31:49.787Z","updated_at":"2026-07-09T00:31:50.051Z","avatar_url":"https://github.com/browserstack.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Sandbox SDK Sample App\n\nA runnable, in-browser walkthrough of the **BrowserStack AI SDK**. It exercises the full SDK surface — auth, manual tracing, auto tracing, prompt compile, dataset runs, eval execution, and experiment runs — and streams every SDK call as a live code snippet so you can see exactly what the SDK is doing.\n\nThe same frontend ships with **two interchangeable backends** (Python FastAPI + Node Express). A toggle in the sidebar flips between them so you can see the SDK surface in either language without restarting anything.\n\n---\n\n## Prerequisites\n\nYou'll need the following before starting:\n\n1. **Docker + Docker Compose** installed locally.\n2. A **BrowserStack Sandbox project** at https://evals.browserstack.com.\n3. **API keys** for that project — see \"One-time project setup\" below.\n4. An **OpenAI key** and/or **Anthropic key** — at least one. Used by the chat stages and by the LLM-based evaluators in the workflows.\n5. The **two SDK tarballs** dropped into `sdk-vendor/` — see \"Drop the SDK tarballs in place\" below.\n\n---\n\n## Drop the SDK tarballs in place\n\nBoth backends install the SDK from local tarballs (vendored under `sdk-vendor/`, gitignored). Grab the latest builds — typically distributed as static links such as `https://static.testops.ai/sdk/python/browserstack_ai_sdk-\u003cversion\u003e.tar.gz` and `https://static.testops.ai/sdk/nodejs/browserstack-ai-sdk-\u003cversion\u003e.tar.gz` — and place them at:\n\n```\nsdk-vendor/\n├── browserstack_ai_sdk-\u003cversion\u003e.tar.gz       # Python SDK\n└── browserstack-ai-sdk-\u003cversion\u003e.tar.gz       # Node SDK (also symlinked as .tgz for npm)\n```\n\nThe `python-backend/Dockerfile` and `node-backend/package.json` reference these filenames explicitly. **If your tarball versions differ**, update:\n\n- `python-backend/Dockerfile` — the two `COPY` + `pip install` lines pointing at `browserstack_ai_sdk-\u003cversion\u003e.tar.gz`\n- `node-backend/Dockerfile` — the `COPY` line pointing at `browserstack-ai-sdk-\u003cversion\u003e.tar.gz`\n- `node-backend/package.json` — the `\"@browserstack/ai-sdk\": \"file:../sdk-vendor/browserstack-ai-sdk-\u003cversion\u003e.tgz\"` dependency\n\nIf you're not sure where to get the tarballs, ask the AI SDK team for the latest static link.\n\n---\n\n## One-time project setup in Sandbox\n\nOpen your project at https://evals.browserstack.com and do **both** of these once:\n\n### 1. Generate Sandbox API keys\n\n- Click the **gear icon** → **Project Settings** → **API keys**.\n- Generate a key pair. You'll get a public key (`pk-to-...`) and a secret key (`sk-to-...`).\n- You'll paste these into the app's auth screen.\n\n### 2. Add an LLM connection on the project\n\nThis step is easy to miss and the workflows will fail silently without it (every evaluation comes back as 0).\n\n- Same gear icon → **Project Settings** → **LLM connections**.\n- Add a connection for the model the evaluators will call. The sample app's evaluators are configured for **`openai / gpt-4o-mini`** by default, so add an OpenAI connection there.\n- If you want the workflows to use Anthropic instead, add an Anthropic connection and update the `modelParams` in the workflow code accordingly.\n\n---\n\n## Run it\n\n```bash\ndocker compose up --build\n```\n\nThis spins up two backends side-by-side:\n\n| Service | URL | Runtime |\n|---|---|---|\n| Python backend | http://localhost:8000 | FastAPI on port 8000 |\n| Node backend | http://localhost:3001 | Express on port 3001 |\n\nBoth serve the same frontend at `/`. **Open either URL** in your browser — the UI is identical. The sidebar has a **Backend** toggle (Python / Node) that flips which one your clicks talk to.\n\n---\n\n## Using the app\n\n### 1. Connect\n\nOn the **Sandbox keys** screen, paste:\n\n- **Sandbox public key** (`pk-to-...`)\n- **Sandbox secret key** (`sk-to-...`)\n- **At least one** of: OpenAI key (`sk-...`) or Anthropic key (`sk-ant-...`)\n\nClick **Validate \u0026 continue**. The app calls `experiments.list()` against your project to verify the keys and discover your project ID.\n\nKeys are kept in browser memory only — refresh and you'll be re-prompted. They're never sent anywhere except `evals.browserstack.com`.\n\n### 2. Tracing\n\nTwo pages demonstrating the two tracing modes:\n\n- **Manual tracing** — chat that wraps the LLM call in `client.trace()` / `start_generation()` / `gen.update()` / `gen.end()` / `trace.score()` / `trace.update()`.\n- **Auto tracing** — same chat, but `Observe.init()` instruments the OpenAI/Anthropic SDK call automatically. Pick provider + model from the dropdown.\n\nBoth pages always show the static SDK script on the right and link to your project's traces page in Sandbox.\n\n### 3. Workflows\n\nFour end-to-end workflows. All are **idempotent** — re-running them reuses existing artifacts instead of duplicating.\n\n| Workflow | What it does |\n|---|---|\n| **Prompt Compile** | Get-or-create a templated prompt (`support-reply-generator`), call `Prompt.compile(...)` with user-supplied tone / customer_name / issue, send the compiled prompt to OpenAI. |\n| **Dataset Run** | Get-or-create a dataset (`support-bot-reranker-eval`), create a dataset run, seed 3 items, verify by listing them back. |\n| **Eval Execution** | Look up the `support-quality` evaluator list, walk its `evaluatorConfigs[]`, build the rich payload, run `Evaluate.evaluation_execution.evaluate(...)` for each row in the dataset. |\n| **Experiment Run** | The full 8-phase orchestration — prompt → dataset → tools → evaluators → evaluator-list → experiment → run → subscribe. |\n\nEach workflow shows the live phase progression on the left and the SDK code being executed on the right. Every artifact created comes with a \"View in Sandbox\" link.\n\n---\n\n## Troubleshooting\n\n**\"Authenticated but no experiments found\"** — your project is empty. Either run a workflow first, or create an experiment in Sandbox manually so `experiments.list()` returns something.\n\n**Workflows complete but every score is 0** — your project doesn't have an LLM connection configured. See \"One-time project setup\" step 2 above.\n\n**Tools API returns \"Project not found\"** in experiment-run Phase 4 — known upstream public-API issue with the Tools endpoints. The workflow continues past it; doesn't affect the rest.\n\n**`sdk-vendor/` is empty** or **`docker compose` build fails on `pip install`/`npm install`** — you haven't placed the SDK tarballs yet. See \"Drop the SDK tarballs in place\" above.\n\n---\n\n## Layout\n\n```\n.\n├── docker-compose.yml\n├── python-backend/          FastAPI on :8000\n│   ├── routes/                  auth, chat_manual, chat_auto, workflows\n│   ├── workflows/               prompt_compile, dataset_run, eval_execution, experiment_run\n│   └── services/                sdk_client, snippet_emitter, idempotency, llm\n├── node-backend/            Express on :3001\n│   └── src/                     mirror of python-backend, file-for-file\n├── shared-frontend/         single HTML/CSS/JS UI mounted by both backends\n└── sdk-vendor/              SDK tarballs (gitignored — source separately)\n```\n\nBoth backends expose an identical SSE event shape (`phase-start` → `code-snippet` → `result` → `phase-end` → `done`) so the frontend doesn't care which one it's talking to.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrowserstack%2Fai-sdk-sample-app","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrowserstack%2Fai-sdk-sample-app","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrowserstack%2Fai-sdk-sample-app/lists"}