{"id":51336924,"url":"https://github.com/dpguthrie/mastering-ai-observability-workshop","last_synced_at":"2026-07-05T06:00:30.644Z","repository":{"id":368223582,"uuid":"1283580131","full_name":"dpguthrie/mastering-ai-observability-workshop","owner":"dpguthrie","description":"AIE World's Fair content for the Mastering AI Observability workshop with Braintrust","archived":false,"fork":false,"pushed_at":"2026-06-29T14:38:02.000Z","size":168,"stargazers_count":16,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-02T03:35:03.589Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dpguthrie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":"SUPPORT_WORKFLOW_ISSUE_FACET.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-29T03:51:43.000Z","updated_at":"2026-06-29T19:34:00.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/dpguthrie/mastering-ai-observability-workshop","commit_stats":null,"previous_names":["dpguthrie/mastering-ai-observability-workshop"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/dpguthrie/mastering-ai-observability-workshop","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fmastering-ai-observability-workshop","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fmastering-ai-observability-workshop/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fmastering-ai-observability-workshop/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fmastering-ai-observability-workshop/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dpguthrie","download_url":"https://codeload.github.com/dpguthrie/mastering-ai-observability-workshop/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dpguthrie%2Fmastering-ai-observability-workshop/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35071463,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-03T02:00:05.635Z","response_time":110,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-02T03:32:36.175Z","updated_at":"2026-07-03T04:00:56.958Z","avatar_url":"https://github.com/dpguthrie.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AI Evals World's Fair Support Flywheel\n\nThis repo is the local workshop app for building a trace-backed agent improvement\nflywheel with Braintrust.\n\n## Start Here\n\n```bash\nuv sync --extra dev\ncp .env.example .env\nuv run python scripts/seed_db.py\nmake ready\nmake chat-ui\n```\n\nOpen \u003chttp://127.0.0.1:8765\u003e.\n\nLive model calls use Braintrust Gateway and require `BRAINTRUST_API_KEY` plus\n`AGENT_DEFAULT_MODEL` in `.env`. Pick a Gateway-configured chat model that\nsupports tool/function calling; `make ready` runs a live support-agent tool-call\nsmoke before the workshop flow. `JUDGE_MODEL` is optional and falls back to\n`AGENT_DEFAULT_MODEL`.\n\n## Core Commands\n\n```bash\nmake create-eval-dataset  # seed Braintrust eval dataset from evals/cases.jsonl\nmake eval              # run the eval from the Braintrust dataset\nmake configure-topics  # enable the workshop Topics facets\nmake push-scorers      # push hosted scorer definitions to Braintrust\nmake import-sample-traces  # import the shared sanitized trace bundle\nmake draft-cases       # optional: create review-draft dataset rows from traces\nmake ready             # check env, database, catalog hint, and live model tool calls\nmake ready-skip-model  # check local readiness without the live model call\nmake traces            # dynamically write production-like traces to Braintrust\nmake traces-reset      # reseed before writing production-like traces\nmake test              # unit and web smoke tests\n```\n\nThe starter eval cases in `evals/cases.jsonl` use Braintrust dataset row shape:\n`input`, optional `expected`, and `metadata`. `make eval` reads from the\nBraintrust dataset named by `EVAL_DATASET`.\n\nSeed data lives in `data/seed.json`. Eval and trace commands reuse an existing\nvalid local DB by default; use the reset commands when you want a clean seed.\n\nTo try a different model for one eval:\n\n```bash\nAGENT_DEFAULT_MODEL=gpt-4o bt eval evals/eval_support_agent.py --no-input\n```\n\n## Workshop Guide\n\n- [WORKSHOP.md](WORKSHOP.md) is the single public workshop guide.\n- [SUPPORT_WORKFLOW_ISSUE_FACET.md](SUPPORT_WORKFLOW_ISSUE_FACET.md) contains the custom Topics facet prompt.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpguthrie%2Fmastering-ai-observability-workshop","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdpguthrie%2Fmastering-ai-observability-workshop","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdpguthrie%2Fmastering-ai-observability-workshop/lists"}