{"id":51437685,"url":"https://github.com/langchain-samples/langsmith-guided-tour","last_synced_at":"2026-07-05T08:30:28.000Z","repository":{"id":368250468,"uuid":"1250707898","full_name":"langchain-samples/langsmith-guided-tour","owner":"langchain-samples","description":"Self-directed Jupyter notebooks for engineers evaluating LangSmith during a POC. The modules cover the full agent engineering loop — build, trace, evaluate, deploy, and surface failure modes — against a single example agent.","archived":false,"fork":false,"pushed_at":"2026-06-29T17:40:37.000Z","size":8633,"stargazers_count":0,"open_issues_count":11,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-29T19:10:55.679Z","etag":null,"topics":["beginner","evaluation","langsmith","observability","workshop"],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/langchain-samples.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-26T22:33:12.000Z","updated_at":"2026-06-29T17:11:34.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/langchain-samples/langsmith-guided-tour","commit_stats":null,"previous_names":["langchain-samples/langsmith-guided-tour"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/langchain-samples/langsmith-guided-tour","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langchain-samples%2Flangsmith-guided-tour","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langchain-samples%2Flangsmith-guided-tour/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langchain-samples%2Flangsmith-guided-tour/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langchain-samples%2Flangsmith-guided-tour/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/langchain-samples","download_url":"https://codeload.github.com/langchain-samples/langsmith-guided-tour/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/langchain-samples%2Flangsmith-guided-tour/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35148606,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-05T02:00:06.290Z","response_time":100,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beginner","evaluation","langsmith","observability","workshop"],"created_at":"2026-07-05T08:30:27.436Z","updated_at":"2026-07-05T08:30:27.979Z","avatar_url":"https://github.com/langchain-samples.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LangSmith POC Modules\n\nSelf-directed Jupyter notebooks for engineers evaluating LangSmith during a POC. The modules cover the full agent engineering loop — build, trace, evaluate, deploy, and surface failure modes — against a single example agent.\n\n## The Modules\n\n| # | Module | Notebook | Duration |\n|---|--------|----------|----------|\n| **00** | **Setup** — env, keys, service verification | [`modules/00_setup.ipynb`](modules/00_setup.ipynb) | ~10 min |\n| **01** | **Build a Deep Agent** — harness, tools, subagents, backends, middleware, HITL, AGENTS.md, skills (optional) | [`modules/01_build_a_deep_agent_optional.ipynb`](modules/01_build_a_deep_agent_optional.ipynb) | ~45 min |\n| **02** | **Tracing** — generate traces and query them with `list_runs` + filter DSL | [`modules/02_tracing.ipynb`](modules/02_tracing.ipynb) | ~20 min |\n| **03** | **Finding Failure Modes** — Chat, Insights Agent, and Engine | [`modules/03_finding_failure_modes.ipynb`](modules/03_finding_failure_modes.ipynb) | ~30 min |\n| **04** | **Datasets and Experiments** — offline evaluation: final-response, single-step, trajectory | [`modules/04_datasets_and_experiments.ipynb`](modules/04_datasets_and_experiments.ipynb) | ~30 min |\n| **05** | **Online Evaluations** — LLM-as-judge run rules that score new traces automatically | [`modules/05_online_evals.ipynb`](modules/05_online_evals.ipynb) | ~25 min |\n| **06** | **Annotation Queues** — route low-scoring runs to human review | [`modules/06_annotation_queues.ipynb`](modules/06_annotation_queues.ipynb) | ~20 min |\n| **07** | **Deploy + Govern** — apply workspace-level gateway policies and ship the agent via LangSmith Deployments (optional) | [`modules/07_deploy_and_govern_optional.ipynb`](modules/07_deploy_and_govern_optional.ipynb) | ~25 min |\n\nModules are designed to run in order. The full sequence is ~3.5 hours; the required-only path (skipping 01 and 07) is ~2 hours.\n\n**Optional modules** are tagged `_optional` in the filename:\n- **Module 01** introduces the `deepagents` framework from scratch. Skip if already familiar with custom tools, subagents, and prompts.\n- **Module 07** covers deployment via LangSmith. Skip if you don't have deployment permissions or are using LangSmith strictly for observability and evaluations.\n\nThe remaining modules form the core observability + evaluation loop.\n\n## Prerequisites\n\n- Python 3.11+\n- [uv](https://docs.astral.sh/uv/getting-started/installation/) (recommended) or pip\n- A LangSmith account ([sign up](https://smith.langchain.com))\n- An API key from your model provider (Anthropic by default; OpenAI, Azure OpenAI, and AWS Bedrock are also supported — see *Switching Models* below)\n- A Tavily API key for the web search tool ([get one](https://tavily.com))\n\n## Setup\n\nModule 00 walks through this end-to-end with verification cells. The short version:\n\n```bash\n# 1. Install dependencies\nuv sync\n\n# 2. Create your .env file\ncp .env.example .env\n# Edit .env and fill in your keys\n\n# 3. Start Jupyter\nuv run jupyter notebook\n```\n\nThen open `modules/00_setup.ipynb` and run the cells in order to verify Python, dependencies, and credentials.\n\n| Key | Required for | Where to get one |\n|---|---|---|\n| `ANTHROPIC_API_KEY` | Modules 01–07 (default model provider) | \u003chttps://console.anthropic.com\u003e |\n| `LANGSMITH_API_KEY` | All modules (tracing + evaluations) | \u003chttps://smith.langchain.com\u003e |\n| `TAVILY_API_KEY` | Modules 01–06 (web search tool used by the agent) | \u003chttps://tavily.com\u003e |\n\nModule 06 (Deploy) additionally requires a LangSmith **service key** (`lsv2_sk_...`), not a personal access token, for deployment permissions.\n\n## Switching Models\n\nAll modules import `model` from `utils/models.py`. Change one line there to swap providers — no notebook edits required.\n\n```python\n# utils/models.py\n\n# Anthropic (default)\nmodel = init_chat_model(\"anthropic:claude-sonnet-4-6\")\n\n# OpenAI\n# model = init_chat_model(\"openai:gpt-4.1-mini\")\n\n# Azure OpenAI\n# from langchain_openai import AzureChatOpenAI\n# model = AzureChatOpenAI(azure_deployment=\"gpt-4.1-mini\", streaming=True)\n\n# AWS Bedrock\n# from langchain_aws import ChatBedrockConverse\n# model = ChatBedrockConverse(provider=\"anthropic\", model_id=\"...\")\n```\n\nThen set the matching API key environment variable in `.env`. See `.env.example` for the full set of supported provider variables.\n\n## Deploy + Govern (Module 07)\n\nModule 07 covers two things: wiring up the LangSmith LLM Gateway with a workspace-level PII/secrets policy, then deploying the governed agent to LangSmith Deployments using the `langgraph` CLI (installed by `uv sync`). The deploy config is `langgraph.json` at the repo root. Two graphs are registered: `client_research` (the primary deployable) and `base_research_agent` (a second example for inspection).\n\nYour `LANGSMITH_API_KEY` must have deployment permissions — use a service key (`lsv2_sk_...`), not a personal access token. The gateway sections require `LANGSMITH_API_KEY_GATEWAY` (same value) and `WORKSPACE_ID` — see `.env.example`.\n\n## Project Structure\n\n```\nlangsmith-guided-tour/\n├── README.md                                  (this file)\n├── pyproject.toml                             (shared dependencies)\n├── .env.example\n├── langgraph.json                             (registers deployable graphs)\n├── utils/\n│   ├── config.py                              (active agent + project name — single source of truth)\n│   ├── models.py                              (model initialization — swap providers here)\n│   ├── search.py                              (resilient Tavily wrapper with canned fallbacks)\n│   └── langsmith_rules.py                     (helpers for run rules + annotation queues)\n├── agents/\n│   ├── client_research_agent.py               (eval-safe agent imported by Modules 02–05 via utils.config)\n│   └── deployable_agents/\n│       ├── client_research/                   (deployable variant — AGENTS.md, skills, CompositeBackend)\n│       │   ├── agent.py\n│       │   ├── AGENTS.md\n│       │   ├── deepagents.toml\n│       │   └── skills/\n│       │       ├── client-brief/SKILL.md\n│       │       └── portfolio-update/SKILL.md\n│       └── base_research_agent/               (second deployable, kept as reference)\n│           ├── agent.py\n│           ├── AGENTS.md\n│           ├── deepagents.toml\n│           └── skills/\n├── images/                                    (diagrams + screenshots referenced by the notebooks)\n├── modules/\n│   ├── 00_setup.ipynb\n│   ├── 01_build_a_deep_agent_optional.ipynb\n│   ├── 02_tracing.ipynb\n│   ├── 03_finding_failure_modes.ipynb\n│   ├── 04_datasets_and_experiments.ipynb\n│   ├── 05_online_evals.ipynb\n│   ├── 06_annotation_queues.ipynb\n│   └── 07_deploy_and_govern_optional.ipynb\n└── skills/\n    └── customize-poc/                         (Claude Code skill for adapting this repo to a new domain)\n        ├── SKILL.md\n        └── notebook-customization-guide.md\n```\n\n## Customizing for a New Domain\n\nThe repo ships specialized for a client research use case. To adapt it for a different industry or use case, the `customize-poc` skill at `skills/customize-poc/` walks a coding agent (Claude Code, for example) through seven structured discovery questions, then executes the end-to-end customization across the agent code, configuration, and all eight notebook modules.\n\n### Workflow\n\n1. **Clone the repo:**\n   ```bash\n   git clone https://github.com/langchain-samples/langsmith-guided-tour.git\n   cd langsmith-guided-tour\n   ```\n\n2. **Create a branch for your variant.** Use the `examples/\u003cvertical\u003e` naming convention (e.g., `examples/insurance-claims`, `examples/legal-contracts`):\n   ```bash\n   git checkout -b examples/\u003cyour-vertical\u003e\n   ```\n\n3. **Open the repo in a coding agent** and invoke the `customize-poc` skill. The skill auto-loads from `.claude/skills/customize-poc/` in any Claude Code session opened on this repo — start the session, then ask the agent to invoke `customize-poc`.\n\n4. **Answer the discovery questions.** The skill asks seven structured follow-ups one at a time (persona, tools, demo data, example queries, eval criteria, deployable identity, skills). Three approval checkpoints — after the spec, after the agent code, before the dataset — catch misunderstandings before they propagate through the notebooks.\n\n5. **Review the output.** The skill runs validation at the end (import probes, notebook syntax checks, residual-content greps). Spot-check a few notebook cells for tone and accuracy before committing.\n\n6. **Commit and push:**\n   ```bash\n   git add -A\n   git commit -m \"Add \u003cyour-vertical\u003e variant\"\n   git push origin examples/\u003cyour-vertical\u003e\n   ```\n\nTo contribute a new example back to the samples repo, open a PR against `main`. To keep the variant private (customer-specific work, internal POCs), fork this repo into your own org first and push the branch there.\n\n## Common Issues\n\n**`langgraph deploy` fails with 403 / permission denied**\nYour API key is a personal access token. Generate a service key (`lsv2_sk_...`) in LangSmith **Settings → Organizations → Access and Security → API Keys**.\n\n**Notebook can't find `utils` / `agents`**\nEach module's setup cell prepends the repo root to `sys.path`. If you moved a notebook, update the `Path().resolve().parent` line to point at the repo root.\n\n**Anthropic API: `tool_use ids were found without tool_result blocks immediately after`**\nThis appears if you submit a regular message to the deployed agent in Studio while a HITL interrupt is pending. The deployable variant in this repo ships without HITL — but if you re-add `interrupt_on={...}` to `agents/deployable_agents/client_research/agent.py`, send the resume command as a `Command(resume=...)` payload rather than plain text.\n\n**Chat (Module 07) unavailable**\nThe in-workspace AI assistant requires a model provider API key configured as a workspace secret in LangSmith **Settings**. Configure one before invoking Chat with `Cmd+I` / `Ctrl+I`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flangchain-samples%2Flangsmith-guided-tour","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flangchain-samples%2Flangsmith-guided-tour","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flangchain-samples%2Flangsmith-guided-tour/lists"}