{"id":34524865,"url":"https://github.com/dpguthrie/cloudflare-example","last_synced_at":"2026-05-30T10:31:14.635Z","repository":{"id":319189297,"uuid":"1077907136","full_name":"dpguthrie/cloudflare-example","owner":"dpguthrie","description":null,"archived":false,"fork":false,"pushed_at":"2025-10-29T13:04:29.000Z","size":1088,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-25T16:28:47.191Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dpguthrie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-16T23:23:28.000Z","updated_at":"2025-10-29T13:04:33.000Z","dependencies_parsed_at":"2025-10-18T11:10:32.028Z","dependency_job_id":"9a21890c-29d9-4f40-937d-480872e392c5","html_url":"https://github.com/dpguthrie/cloudflare-example","commit_stats":null,"previous_names":["dpguthrie/cloudflare-example"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dpguthrie/cloudflare-example","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fcloudflare-example","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fcloudflare-example/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fcloudflare-example/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fcloudflare-example/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dpguthrie","download_url":"https://codeload.github.com/dpguthrie/cloudflare-example/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fcloudflare-example/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33689564,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-30T02:00:06.278Z","response_time":92,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-12-24T05:07:00.024Z","updated_at":"2026-05-30T10:31:14.606Z","avatar_url":"https://github.com/dpguthrie.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Braintrust SDK on Cloudflare Workers\n\nExample implementation showing how to run Braintrust evaluations and experiments on Cloudflare Workers using two different approaches.\n\n## Quick Start\n\n```bash\n# Install dependencies\nnpm install\n\n# Set up local environment\necho \"BRAINTRUST_API_KEY=your-key\" \u003e .dev.vars\necho \"OPENAI_API_KEY=your-key\" \u003e\u003e .dev.vars\n\n# Run locally\nnpm run dev\n\n# Test the endpoints\ncurl http://localhost:8787/trace              # Tracing example\ncurl http://localhost:8787/run-eval           # Eval() approach\ncurl http://localhost:8787/run-experiment     # Logging SDK\ncurl http://localhost:8787/run-direct-api     # Direct API\n\n# Deploy to production\nnpm run deploy\n```\n\n## Four Examples\n\nThis example demonstrates **four ways** to use Braintrust on Cloudflare Workers:\n\n### Evaluation \u0026 Experimentation:\n1. **Eval()** - High-level SDK for evaluations\n2. **Logging SDK** - Full control with init + traced + log (✅ **WORKS in Vitest** with deps.optimizer!)\n3. **Direct API** - No SDK, pure REST API calls (✅ **WORKS in Vitest**)\n\n### Observability:\n4. **Tracing Example** - Real-time tracing with OpenAI tool calls\n\n**🎯 Want to use Braintrust with Vitest?** See [docs/VITEST_SOLUTION.md](./docs/VITEST_SOLUTION.md) for the complete guide.\n\n**❓ Why doesn't Eval() work in Vitest?** See [docs/WHY_EVAL_DOESNT_WORK.md](./docs/WHY_EVAL_DOESNT_WORK.md) for the technical explanation.\n\n### Approach 1: Eval() Function\n\n**Best for:** Simple, standard evaluations with minimal setup.\n\n**Code:**\n```typescript\nimport { Eval, login } from 'braintrust';\n\nconst result = await Eval(\"my-experiment\", {\n  data: () =\u003e [\n    { input: \"What is 2+2?\", expected: \"4\" },\n  ],\n  task: async (input) =\u003e {\n    // Your LLM call\n    return await callOpenAI(input);\n  },\n  scores: [\n    (output, expected) =\u003e ({\n      name: \"contains_expected\",\n      score: output.includes(expected.expected) ? 1 : 0,\n    }),\n  ],\n});\n```\n\n**Endpoints:**\n- Production: `https://your-worker.workers.dev/run-eval`\n- Local: `http://localhost:8787/run-eval`\n\n**Implementation:** See `src/index.ts:208-265`\n\n### Approach 2: Logging SDK (init + traced + log)\n\n**Best for:** Custom evaluation logic, framework integration, more control.\n\n**Features:**\n- ✅ Automatic nested tracing with `wrapOpenAI()`\n- ✅ Full control over execution flow\n- ✅ Custom metadata and scoring\n\n**Code:**\n```typescript\nimport { init, login, wrapOpenAI } from 'braintrust';\nimport OpenAI from 'openai';\n\n// Wrap OpenAI for automatic nested tracing\nconst client = wrapOpenAI(new OpenAI({ apiKey: env.OPENAI_API_KEY }));\n\n// Initialize experiment\nconst experiment = init({\n  project: \"my-project\",\n  experiment: \"my-experiment\",\n  apiKey: env.BRAINTRUST_API_KEY,\n});\n\n// Run test cases\nfor (const { input, expected } of dataset) {\n  await experiment.traced(async (span) =\u003e {\n    // wrapOpenAI automatically creates nested spans for API calls\n    const response = await client.chat.completions.create({\n      model: \"gpt-4o-mini\",\n      messages: [{ role: \"user\", content: input }],\n    });\n\n    const output = response.choices[0].message.content;\n    const score = output.includes(expected) ? 1 : 0;\n\n    span.log({\n      input,\n      output,\n      expected,\n      scores: { contains_expected: score },\n      metadata: { model: response.model, usage: response.usage },\n    });\n  });\n}\n\nconst summary = await experiment.summarize();\n```\n\n**Endpoints:**\n- Production: `https://your-worker.workers.dev/run-experiment`\n- Local: `http://localhost:8787/run-experiment`\n\n**Implementation:** See `src/index.ts:280-365`\n\n### Approach 3: Direct REST API + OpenTelemetry ✨\n\n**Best for:** Testing with Vitest, maximum control, no SDK dependencies.\n\n**Key Advantage:** ✅ **Can be directly imported and tested in Vitest!**\n\n**Code:**\n```typescript\nimport { runExperimentWithDirectAPI } from './direct-api';\n\n// Create experiment using REST API\nconst result = await runExperimentWithDirectAPI(env);\n\n// Insert events\nawait insertExperimentEvents(apiKey, experimentId, [\n  {\n    input: 'What is 2+2?',\n    output: '4',\n    expected: '4',\n    scores: { correctness: 1 },\n  },\n]);\n\n// Get summary\nconst summary = await summarizeExperiment(apiKey, experimentId);\n```\n\n**Endpoints:**\n- Production: `https://your-worker.workers.dev/run-direct-api`\n- Local: `http://localhost:8787/run-direct-api`\n\n**Implementation:** See `src/direct-api.ts`\n\n**Vitest Testing:**\n```typescript\n// ✅ This WORKS - direct import in Vitest!\nimport { runExperimentWithDirectAPI } from './direct-api';\n\ntest('my test', async () =\u003e {\n  const result = await runExperimentWithDirectAPI(env);\n  expect(result.summary).toBeDefined();\n});\n```\n\n**APIs Used:**\n- [Create Experiment](https://www.braintrust.dev/docs/reference/api/Experiments#create-experiment)\n- [Insert Events](https://www.braintrust.dev/docs/reference/api/Experiments#insert-experiment-events)\n- [Summarize](https://www.braintrust.dev/docs/reference/api/Experiments#summarize-experiment)\n- [OpenTelemetry Integration](https://www.braintrust.dev/docs/integrations/opentelemetry)\n\n### Approach 4: Real-Time Tracing with Tool Calls\n\n**Best for:** Observability, monitoring LLM applications, debugging tool calling behavior.\n\n**Key Advantage:** ✅ **Real-time tracing of OpenAI tool calls with automatic nested spans!**\n\n**Code:**\n```typescript\nimport { initLogger, wrapOpenAI, wrapTraced } from 'braintrust';\n\n// Initialize logger with asyncFlush\nconst logger = initLogger({\n  projectName: \"my-app\",\n  apiKey: env.BRAINTRUST_API_KEY,\n  asyncFlush: true,\n});\n\n// Wrap OpenAI for automatic tracing\nconst client = wrapOpenAI(new OpenAI({ apiKey: env.OPENAI_API_KEY }));\n\n// Wrap tool functions for tracing\nconst getCurrentWeather = wrapTraced(async (location: string) =\u003e {\n  // Your tool logic\n  return JSON.stringify({ location, temperature: 68 });\n}, { name: \"getCurrentWeather\" });\n\n// Trace the entire operation\nawait logger.traced(async (span) =\u003e {\n  // OpenAI calls are automatically traced\n  const response = await client.chat.completions.create({\n    model: \"gpt-4o-mini\",\n    messages: [{ role: \"user\", content: \"What's the weather?\" }],\n    tools: [/* tool definitions */],\n  });\n\n  // Tool calls are traced when you invoke them\n  const result = await getCurrentWeather(\"San Francisco\");\n\n  span.log({ input, output: result });\n});\n\n// Ensure traces are flushed (important for Workers!)\nctx.waitUntil(logger.flush());\n```\n\n**Endpoints:**\n- Production: `https://your-worker.workers.dev/trace`\n- Local: `http://localhost:8787/trace`\n\n**Implementation:** See `src/index.ts:10-206`\n\n**Features:**\n- ✅ Automatic nested tracing for OpenAI calls\n- ✅ Tool call tracing with wrapTraced()\n- ✅ Async flush for Workers compatibility\n- ✅ Real-time observability in Braintrust dashboard\n\n### Which Should You Use?\n\n| Feature | Eval() | Logging SDK | Direct API | Tracing |\n|---------|--------|-------------|------------|---------|\n| **Setup** | Minimal | More code | More code | Minimal |\n| **Control** | Framework-managed | Full control | Full control | Full control |\n| **Nested tracing** | Automatic | Automatic (wrapOpenAI) | Manual (OTEL) | Automatic |\n| **Best for** | Standard evals | Custom evals/experiments | Testing in Vitest | Observability |\n| **Learning curve** | Easy | Moderate | Moderate | Easy |\n| **Vitest direct import** | ❌ No | ✅ **YES!** (deps.optimizer) | ✅ **YES!** | ✅ **YES!** (deps.optimizer) |\n| **Production use** | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes |\n| **SDK dependency** | ✅ Yes | ✅ Yes | ❌ No | ✅ Yes |\n\n**See [docs/APPROACHES_COMPARISON.md](./docs/APPROACHES_COMPARISON.md) for detailed comparison.**\n\n## Setup\n\n### 1. Install Dependencies\n\n```bash\nnpm install\n```\n\n### 2. Configure API Keys\n\n#### For Local Development\n\nCreate a `.dev.vars` file (don't commit this!):\n\n```bash\necho \"BRAINTRUST_API_KEY=your-braintrust-key\" \u003e .dev.vars\necho \"OPENAI_API_KEY=your-openai-key\" \u003e\u003e .dev.vars\n```\n\nGet your Braintrust API key from: https://www.braintrust.dev/app/settings?subroute=api-keys\n\n#### For Production\n\nSet secrets in Cloudflare:\n\n```bash\nnpx wrangler secret put BRAINTRUST_API_KEY\nnpx wrangler secret put OPENAI_API_KEY\n```\n\n### 3. Enable Node.js Compatibility\n\nAlready configured in `wrangler.toml`:\n\n```toml\ncompatibility_flags = [\"nodejs_compat\"]\n```\n\nThis is **required** for the Braintrust SDK to work on Cloudflare Workers.\n\n## Local Development\n\n### Start the Dev Server\n\n```bash\nnpm run dev\n```\n\nThis starts a local server at `http://localhost:8787`.\n\n### Test All Endpoints\n\n**Important:** Local dev uses `http://` (not `https://`)\n\n```bash\n# Tracing example\ncurl http://localhost:8787/trace\n\n# Eval() approach\ncurl http://localhost:8787/run-eval\n\n# Logging SDK approach\ncurl http://localhost:8787/run-experiment\n\n# Direct API approach\ncurl http://localhost:8787/run-direct-api\n\n# View available endpoints\ncurl http://localhost:8787/\n```\n\n**Common mistake:** Using `https://localhost:8787` will fail. Local dev is HTTP only.\n\n### Watch for Changes\n\nThe dev server automatically reloads when you edit `src/index.ts`.\n\n## Deployment\n\n### Deploy to Production\n\n```bash\nnpm run deploy\n```\n\nYou'll get a URL like: `https://braintrust-eval-worker.your-subdomain.workers.dev`\n\n### Test Production\n\n```bash\ncurl https://your-worker.workers.dev/trace\ncurl https://your-worker.workers.dev/run-eval\ncurl https://your-worker.workers.dev/run-experiment\ncurl https://your-worker.workers.dev/run-direct-api\n```\n\n### Schedule Automatic Runs\n\nEdit `wrangler.toml` to enable cron triggers:\n\n```toml\n[triggers]\ncrons = [\"0 0 * * *\"]  # Daily at midnight UTC\n```\n\nThen redeploy:\n\n```bash\nnpm run deploy\n```\n\n**Cron examples:**\n- `\"0 0 * * *\"` - Daily at midnight\n- `\"0 */6 * * *\"` - Every 6 hours\n- `\"*/15 * * * *\"` - Every 15 minutes\n\n## Testing\n\n### Run Integration Tests\n\n```bash\nnpm test\n```\n\nOutput:\n```\n✓ test/integration/worker.test.ts (4 tests) 42ms\n  ✓ should respond to fetch requests\n  ✓ should have run-eval endpoint available\n  ✓ should be able to run eval with proper environment\n  ✓ should run experiment using logging SDK\n\nTest Files  1 passed (1)\n     Tests  4 passed (4)\n```\n\n### Vitest Testing\n\n#### ✅ Approaches that Work with Direct Imports (RECOMMENDED)\n\n**Logging SDK and Tracing** work perfectly with `deps.optimizer` configuration!\n\n```typescript\n// ✅ This WORKS with deps.optimizer!\nimport { init, login, wrapOpenAI, initLogger } from 'braintrust';\n\ntest('logging SDK test', async () =\u003e {\n  await login({ apiKey: env.BRAINTRUST_API_KEY });\n\n  const experiment = init({\n    project: 'test',\n    experiment: 'my-test'\n  });\n\n  await experiment.traced(async (span) =\u003e {\n    span.log({ input: 'test', output: 'result', scores: { score: 1 } });\n  });\n\n  const summary = await experiment.summarize();\n  expect(summary).toBeDefined();\n});\n```\n\n**Direct API** also works with direct imports:\n\n```typescript\n// ✅ Direct API approach works!\nimport { runExperimentWithDirectAPI } from './direct-api';\n\ntest('direct API test', async () =\u003e {\n  const result = await runExperimentWithDirectAPI(env);\n  expect(result.summary).toBeDefined();\n});\n```\n\n**Configuration Required:** See `test/integration/vitest.config.ts` for the `deps.optimizer` setup.\n\nSee working examples:\n- `test/integration/logging-sdk-optimizer.test.ts` - Logging SDK with deps.optimizer\n- `test/integration/direct-import-works.test.ts` - Direct API approach\n\n#### ❌ Eval() Does Not Work with Direct Imports\n\n**Eval()** cannot be directly imported in Vitest tests:\n\n```typescript\n// ❌ This does NOT work\nimport { Eval } from 'braintrust';\n\ntest('eval test', async () =\u003e {\n  await Eval('test', { ... }); // Error: Eval is not exported\n});\n```\n\n**Why:** Eval() is not exported in the browser/edge build of the Braintrust SDK. It requires Node.js-specific features not available in Cloudflare Workers.\n\n**Solution for Eval():** Test via HTTP endpoints\n\n```typescript\n// ✅ This WORKS for Eval()\nimport { SELF } from 'cloudflare:test';\n\ntest('eval via HTTP', async () =\u003e {\n  const response = await SELF.fetch('http://example.com/run-eval');\n  expect(response.status).toBe(200);\n});\n```\n\n**See [test/README.md](./test/README.md) for testing details.**\n\n## Customization\n\nEdit `src/index.ts` to customize:\n\n### Change the Dataset\n\n```typescript\nconst dataset = [\n  { input: \"Your question here\", expected: \"Expected answer\" },\n  // Add more test cases\n];\n```\n\n### Change the Model\n\n```typescript\nmodel: \"gpt-4o\",  // or gpt-4, gpt-3.5-turbo, etc.\n```\n\n### Change Scoring\n\n```typescript\n// Custom scoring logic\nconst score = myCustomScoringFunction(output, expected);\n\nspan.log({\n  scores: {\n    custom_metric: score,\n    another_metric: anotherScore,\n  },\n});\n```\n\n### Use Different LLM Providers\n\nReplace the OpenAI call with any other provider:\n\n```typescript\n// Anthropic\nconst response = await fetch('https://api.anthropic.com/v1/messages', ...);\n\n// Gemini\nconst response = await fetch('https://generativelanguage.googleapis.com/v1/...', ...);\n\n// Or any other API\n```\n\n## Project Structure\n\n```\ncloudflare-example/\n├── src/\n│   └── index.ts              # Worker code (both approaches)\n├── test/\n│   ├── integration/\n│   │   ├── vitest.config.ts  # Test configuration\n│   │   └── worker.test.ts    # Integration tests\n│   └── README.md             # Testing guide\n├── docs/\n│   └── APPROACHES_COMPARISON.md  # Detailed comparison\n├── wrangler.toml             # Worker configuration\n├── package.json              # Dependencies and scripts\n└── README.md                 # This file\n```\n\n## Monitoring\n\n### View Logs\n\n1. Go to https://dash.cloudflare.com\n2. Select \"Workers \u0026 Pages\"\n3. Click on your worker\n4. View real-time logs\n\n### View Results in Braintrust\n\nAfter running an evaluation, visit the experiment URL:\n\n```\nhttps://www.braintrust.dev/app/projects/{project}/experiments/{id}\n```\n\nThe URL is returned in the response from both endpoints.\n\n## Troubleshooting\n\n### \"Error: No API key found\"\n\n**Solution:** Set your API keys (see Setup section above).\n\n### \"Worker exceeded CPU time limit\"\n\n**Cause:** Free tier has 10ms CPU time limit.\n\n**Solutions:**\n- Upgrade to paid tier ($5/mo) for 30s limit\n- Reduce number of test cases\n- Use faster models\n\n### \"Module not found\"\n\n**Solution:** Run `npm install` to install dependencies.\n\n### Tests fail with \"No such module 'node:os'\"\n\n**This is expected** if trying to import Braintrust SDK directly in tests.\n\n**Solution:** Use integration tests via HTTP (already set up in this repo).\n\n### Local dev not working\n\n**Check:**\n1. `.dev.vars` file exists with valid API keys\n2. Port 8787 is not in use\n3. Run `npm install` first\n\n## FAQ\n\n**Q: Does Braintrust Eval() work on Cloudflare Workers?**\nA: Yes! Both Eval() and the Logging SDK work perfectly with `nodejs_compat` enabled.\n\n**Q: Can I test Braintrust code with Vitest?**\nA: Yes, but only via integration tests (HTTP endpoints), not direct imports. See Testing section above.\n\n**Q: Which approach should I use?**\nA: Use Eval() for simple cases, Logging SDK for custom logic. Both work equally well on Workers.\n\n**Q: Can I use other AI providers besides OpenAI?**\nA: Yes! Replace the API call with any provider (Anthropic, Gemini, etc.).\n\n**Q: How do I add more test cases?**\nA: Edit the `dataset` array in `src/index.ts`.\n\n## Resources\n\n- [Braintrust Eval() Docs](https://www.braintrust.dev/docs/guides/evals)\n- [Braintrust Logging SDK Docs](https://www.braintrust.dev/docs/platform/experiments/write#logging-sdk)\n- [Cloudflare Workers Docs](https://developers.cloudflare.com/workers/)\n- [Wrangler CLI Docs](https://developers.cloudflare.com/workers/wrangler/)\n- [Node.js Compatibility](https://developers.cloudflare.com/workers/runtime-apis/nodejs/)\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpguthrie%2Fcloudflare-example","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdpguthrie%2Fcloudflare-example","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpguthrie%2Fcloudflare-example/lists"}