{"id":51500118,"url":"https://github.com/ginaecho/session-typed-skills","last_synced_at":"2026-07-07T17:30:47.412Z","repository":{"id":369024121,"uuid":"1286818348","full_name":"ginaecho/session-typed-skills","owner":"ginaecho","description":"Statically type check and ensure interaction safety among agents through compiled skills","archived":false,"fork":false,"pushed_at":"2026-07-03T09:14:41.000Z","size":3943,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-03T09:19:41.526Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ginaecho.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"ROADMAP.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-07-02T06:02:13.000Z","updated_at":"2026-07-03T09:14:45.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ginaecho/session-typed-skills","commit_stats":null,"previous_names":["ginaecho/session-typed-skills"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/ginaecho/session-typed-skills","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ginaecho%2Fsession-typed-skills","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ginaecho%2Fsession-typed-skills/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ginaecho%2Fsession-typed-skills/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ginaecho%2Fsession-typed-skills/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ginaecho","download_url":"https://codeload.github.com/ginaecho/session-typed-skills/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ginaecho%2Fsession-typed-skills/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35237753,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-07T02:00:07.222Z","response_time":90,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-07T17:30:43.241Z","updated_at":"2026-07-07T17:30:47.403Z","avatar_url":"https://github.com/ginaecho.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Session-Typed Judge Panel (STJP) — Quick Start \u0026 Running Experiments\n\n**A session-typed static compiler for safe interactions in multi-agent systems.**\n\nMulti-agent systems fail in the spaces *between* agents: one agent acts before authorization, two agents wait forever (deadlock), everyone wastes tokens negotiating coordination. STJP type-checks the **conversation itself** before any agent runs — catching deadlocks, catching unsafe orderings, and compiling safe per-agent prompts with a runtime guard and scheduler.\n\n## 📖 Documentation\n\n**Start here:** Read [`docs/README.md`](docs/README.md) for the full documentation index.\n\n**Quick tours:**\n- **What is STJP?** → [`docs/1_TECH_SETUP.md`](docs/1_TECH_SETUP.md) (15 min)\n- **How do we test it?** → [`docs/2_TESTING_STRATEGIES.md`](docs/2_TESTING_STRATEGIES.md) (20 min)\n- **Latest results** → [`docs/5_RUN_REPORTS_EXPLAINED.md`](docs/5_RUN_REPORTS_EXPLAINED.md) (plain English)\n- **Why safety matters** → [`docs/6_USE_CASE_DEADLOCK_SAFETY.md`](docs/6_USE_CASE_DEADLOCK_SAFETY.md) (real examples)\n\n---\n\n## 🚀 Quick Start\n\n### Prerequisites\n\n- **Python 3.13**\n- **Java 17** (for Scribble compiler)\n- **Azure subscription** with Azure AI Foundry access\n- **Git**\n\n### Setup (5 minutes)\n\n```bash\n# Clone the repo\ngit clone https://github.com/ginaecho/session-typed-skills\ncd session-typed-skills\n\n# Clone Scribble (the static checker)\ngit clone https://github.com/scribble/scribble-java\n\n# Install Python dependencies\npip install -r stjp_core/requirements-core.txt\n\n# Configure Azure\naz login\n# Create .env file (see \"Configuration\" section below)\n```\n\n### Configuration\n\nCreate `stjp_core/.env`:\n\n```\nAZURE_TENANT_ID=\u003cyour-tenant-id\u003e\nAZURE_SUBSCRIPTION_ID=\u003cyour-subscription-id\u003e\nAZURE_RESOURCE_GROUP=\u003cyour-resource-group\u003e\nAZURE_PROJECT_NAME=\u003cyour-ai-foundry-project\u003e\nAZURE_LOCATION=eastus\nOPENAI_API_KEY=\u003cyour-openai-key\u003e  # optional, for non-Foundry models\n```\n\nSee `stjp_core/CLAUDE.md` for detailed setup instructions.\n\n---\n\n## 🧪 Running Experiments\n\n### 1. Run a single case with one arm (5 minutes)\n\nThe simplest way to test STJP:\n\n```bash\ncd experiments\n\n# Run finance case, 1 trial, with the full STJP stack (min_llmvalid_sched)\npython scripts/case_runner.py finance 1 --arms min_llmvalid_sched\n\n# Check results\nls cases/finance/runs/\n```\n\n**Output:**\n- `events_min_llmvalid_sched.jsonl` — every message, in order\n- `summary.json` — metrics (success rate, tokens, violations)\n- `prompts/min_llmvalid_sched/` — the exact prompts each agent saw\n\n### 2. Compare STJP with baseline (10 minutes)\n\nRun intent-only vs the full STJP stack on the same case:\n\n```bash\n# Run 3 trials each\npython scripts/case_runner.py finance 3 \\\n  --arms bare,min_llmvalid_sched\n```\n\n**Output comparison:**\n\n| Metric | bare (intent-only) | min_llmvalid_sched (STJP) |\n|---|---|---|\n| Success rate | Often 0% | 100% |\n| Disasters | Usually many | 0 |\n| Tokens/trial | Highly variable | ~13k (optimized) |\n| Cost-to-goal | ∞ (fails) | 13k/success |\n\n### 3. Run the full 7-arm benchmark (1 hour)\n\nThis runs all arms on a case with n=10 trials each (produces the results in `docs/results/RESULT_4_FULL_STACK.md`):\n\n```bash\n# Run all arms (this is the official benchmark)\npython scripts/case_runner.py finance 10\n\n# View dashboard\npython scripts/index_builder.py\nopen INDEX.html\n```\n\n**The 7 arms tested:**\n- **bare** — intent only (baseline)\n- **maf_groupchat_llmvalid** — global protocol text with orchestrator\n- **global_decentralized** — global protocol text, decentralized\n- **min_llmvalid** — per-agent contract (observer, no enforcement)\n- **spec_llmvalid_gate** — per-agent contract + enforcement gate\n- **min_llmvalid_gate** — lean contract + enforcement gate\n- **min_llmvalid_sched** — lean contract + gate + EFSM scheduler (full STJP)\n\n### 4. Run a different case\n\nAvailable cases: `finance`, `banking`, `trade_deadlock`, `report_pipeline`\n\n```bash\n# Try the deadlock case (tests claim: \"only static check catches deadlock\")\npython scripts/case_runner.py trade_deadlock 3 --arms bare,min_llmvalid_sched\n\n# This should show:\n# - bare: 0% success (agents deadlock forever)\n# - min_llmvalid_sched: 100% success (Scribble caught it before running)\n```\n\n### 5. Run with custom options\n\n```bash\n# Run a subset of arms\npython scripts/case_runner.py finance 5 \\\n  --arms bare,min_llmvalid_gate,min_llmvalid_sched\n\n# Run with semantic goal evaluation (LLM judge for goals)\npython scripts/case_runner.py finance 3 --semantic\n\n# Run and log to file\npython scripts/case_runner.py finance 5 2\u003e\u00261 | tee my_run.log\n\n# See all options\npython scripts/case_runner.py --help\n```\n\n---\n\n## 📊 Understanding the Results\n\n### The summary.json (Set A — Conformance \u0026 Cost)\n\n```json\n{\n  \"case\": \"finance\",\n  \"arm\": \"min_llmvalid_sched\",\n  \"n_trials\": 10,\n  \"succeeded\": 10,\n  \"success_rate_pct\": 100,\n  \"viol_events\": 0,\n  \"total_tokens\": 133140,\n  \"total_seconds\": 320,\n  \"tokens_per_trial\": 13314,\n  \"calls_per_trial\": 11.4,\n  \"cost_to_goal\": 13314\n}\n```\n\n**Key metrics:**\n- **success_rate_pct** — % of trials that finished correctly (0–100%)\n- **viol_events** — how many messages violated the protocol\n- **cost_to_goal** — tokens ÷ success rate (the true cost of delivery)\n\n### The summary_eval.json (Set B — Goal Achievement)\n\n```json\n{\n  \"strict_pct\": 100,\n  \"role_pair_pct\": 100,\n  \"strict_per_goal\": {\n    \"G1_high_revenue_above_50k\": 100,\n    \"G2_audit_if_high\": 100\n  }\n}\n```\n\n**Key metrics:**\n- **strict_pct** — % of trials where all goals achieved (100% = perfect)\n- **role_pair_pct** — relaxed version (any message between roles counts)\n\n### The events file (detailed trace)\n\n```bash\n# View the message trace\ntail -50 cases/finance/runs/LATEST/events_min_llmvalid_sched.jsonl\n\n# Human-readable format\ncat cases/finance/runs/LATEST/events_min_llmvalid_sched.jsonl | jq '.[] | \"\\(.step): \\(.sender) -\u003e \\(.receiver) [\\(.label)] = \\(.payload)\"'\n```\n\nEach line is a message:\n```json\n{\n  \"step\": 3,\n  \"sender\": \"Fetcher\",\n  \"receiver\": \"TaxSpecialist\",\n  \"label\": \"HighRevenue\",\n  \"payload\": \"60000\",\n  \"violation\": null\n}\n```\n\nIf `violation` is not null, the protocol was violated:\n```json\n{\n  \"violation\": {\n    \"type\": \"off_protocol\",\n    \"role\": \"TaxSpecialist\",\n    \"state\": \"state_5\",\n    \"expected\": [\"HighRevenue\"]\n  }\n}\n```\n\n---\n\n## 🆕 Creating a New Use Case\n\nFollow the step-by-step guide in [`docs/4_HOW_TO_CREATE_USE_CASES.md`](docs/4_HOW_TO_CREATE_USE_CASES.md).\n\n**Quick checklist:**\n\n1. **Create case directory:**\n   ```bash\n   mkdir -p experiments/cases/my_case/protocols\n   ```\n\n2. **Write `case.yaml`:**\n   ```yaml\n   case_id: my_case\n   description: My new protocol\n   version: v1\n   protocol_name: MyProtocol\n   roles: [Agent1, Agent2, Agent3]\n   terminal_label: Done\n   intent: |\n     Your task description here\n   goals:\n     - id: G1\n       description: First goal\n       anchor: {sender: Agent1, receiver: Agent2, label: FirstMessage}\n   ```\n\n3. **Write protocol (`protocols/v1.scr`):**\n   ```scribble\n   global protocol MyProtocol(role Agent1, role Agent2, role Agent3) {\n       msg1(String) from Agent1 to Agent2;\n       msg2(String) from Agent2 to Agent3;\n       done(String) from Agent3 to Agent1;\n   }\n   ```\n\n4. **Add refinements (`protocols/v1.refn`, optional):**\n   ```ini\n   [Agent1 -\u003e Agent2 : msg1]\n   type: str\n   require: len(x) \u003e 0\n   ```\n\n5. **Test it:**\n   ```bash\n   python scripts/case_runner.py my_case 3 --arms bare,min_llmvalid_sched\n   ```\n\nSee [`docs/4_HOW_TO_CREATE_USE_CASES.md`](docs/4_HOW_TO_CREATE_USE_CASES.md) for detailed steps.\n\n---\n\n## 🔍 Troubleshooting\n\n### \"Protocol validation failed\"\n\nThe Scribble compiler found a problem (deadlock, unreachable state, inconsistency).\n\n**Fix:**\n1. Read the Scribble error message carefully\n2. Check [`docs/6_USE_CASE_DEADLOCK_SAFETY.md`](docs/6_USE_CASE_DEADLOCK_SAFETY.md) for examples\n3. Revise your protocol and re-run\n\n### \"Agents are getting stuck\"\n\nAgents reach a state where nothing can happen next (liveness failure).\n\n**Causes:**\n- Missing a required message in the protocol\n- Refinement too strict (payloads fail validation)\n- Wrong choice logic\n\n**Debug:**\n```bash\n# View the trace where it got stuck\ncat cases/\u003ccase\u003e/runs/LATEST/events_\u003carm\u003e.jsonl | jq '.[] | select(.step \u003e 20)'\n\n# Check the agent's view at that step\ntail -100 \u003crun_dir\u003e/\u003cagent\u003e_transcript.log\n```\n\n### \"High token usage\"\n\nIf intent-only (bare) uses way more tokens than expected:\n\n- **Expected:** Agents guess, debate coordination, waste tokens (that's the point of the test)\n- **If bare is cheap:** Try a harder case that requires real coordination\n\n### \"Azure authentication failed\"\n\n```bash\n# Verify login\naz account show\n\n# Re-authenticate\naz login --use-device-code\n```\n\n---\n\n## 📁 Project Structure\n\n```\n.\n├── docs/                          # Documentation (start here)\n│   ├── README.md                  # Docs index\n│   ├── 1_TECH_SETUP.md           # Foundation guide\n│   ├── 2_TESTING_STRATEGIES.md   # Testing methodology\n│   ├── 3_BENCHMARK_DESIGN_EXPLAINED.md\n│   ├── 4_HOW_TO_CREATE_USE_CASES.md\n│   ├── 5_RUN_REPORTS_EXPLAINED.md\n│   ├── 6_USE_CASE_DEADLOCK_SAFETY.md\n│   ├── reference/                 # Current technical deep-dives (glossary, Scribble\n│   │                              #   extensions, gate internals, Foundry wiring, v3 plan)\n│   ├── results/                   # Current evidence (latest run report, canonical\n│   │                              #   results, deadlock + token-efficiency demos)\n│   └── archive/                   # Superseded docs (nothing deleted)\n├── experiments/\n│   ├── cases/\n│   │   ├── finance/              # Example case\n│   │   ├── banking/\n│   │   ├── trade_deadlock/\n│   │   └── report_pipeline/\n│   ├── baselines/                # The 7 arms (runners)\n│   ├── scripts/\n│   │   ├── case_runner.py        # Main benchmark driver\n│   │   ├── case_loader.py        # Load case.yaml\n│   │   └── index_builder.py      # Build dashboard\n│   └── INDEX.html                # Results dashboard\n├── stjp_core/                     # Library\n│   ├── compiler/                 # Scribble integration\n│   ├── monitor/                  # Runtime monitor\n│   ├── foundry/                  # Azure integration\n│   ├── generation/               # Skill/prompt generation\n│   └── CLAUDE.md                 # Setup guide\n├── scribble-java/                # Scribble compiler (vendored)\n├── ROADMAP.md                    # Future phases\n└── README.md                     # (you are here)\n```\n\n---\n\n## 🎯 Key Results (2026-07-02, n=10 finance case, GPT-5.4)\n\n| Arm | Success | Disasters | Cost-to-goal | Calls | Speed |\n|---|---|---|---|---|---|\n| bare (intent-only) | 0% | 18 | ∞ | — | — |\n| global text | 100% | 0 | 120k | 41.8 | 124s |\n| local contract | 60% | 0 | 144k | 84.9 | 223s |\n| **STJP (full stack)** | **100%** | **0** | **13.3k** | **11.4** | **32s** |\n\n**The headline:** STJP is 9× cheaper, 4× faster, same safety as global protocol text.\n\nSee [`docs/5_RUN_REPORTS_EXPLAINED.md`](docs/5_RUN_REPORTS_EXPLAINED.md) for the full breakdown.\n\n---\n\n## 📖 Next Steps\n\n**Learn the system:**\n1. Read [`docs/1_TECH_SETUP.md`](docs/1_TECH_SETUP.md) (15 min)\n2. Run a 1-trial test: `python experiments/scripts/case_runner.py finance 1 --arms min_llmvalid_sched`\n3. Read [`docs/5_RUN_REPORTS_EXPLAINED.md`](docs/5_RUN_REPORTS_EXPLAINED.md) to interpret results\n\n**Run the full benchmark:**\n1. Read [`docs/2_TESTING_STRATEGIES.md`](docs/2_TESTING_STRATEGIES.md) (understand fairness)\n2. Run: `python experiments/scripts/case_runner.py finance 10`\n3. View: `python experiments/scripts/index_builder.py \u0026\u0026 open INDEX.html`\n\n**Create your own case:**\n1. Follow [`docs/4_HOW_TO_CREATE_USE_CASES.md`](docs/4_HOW_TO_CREATE_USE_CASES.md)\n2. Add a new directory under `experiments/cases/`\n3. Define protocol, case config, goals\n4. Test with: `python experiments/scripts/case_runner.py \u003ccase_id\u003e 3`\n\n---\n\n## 🤝 Contributing\n\n**New cases:** Add to `experiments/cases/`. Start with a task description, protocol (Scribble), and goals.\n\n**New arms:** Add to `experiments/baselines/registry.py` and `instructions.py`. See `experiments/CLAUDE.md` for the mechanics.\n\n**Theory/roadmap:** See `ROADMAP.md` and `RESEARCH.md`.\n\n---\n\n## ❓ Questions?\n\n- **How do I run STJP?** → This README (you are here)\n- **What is STJP technically?** → [`docs/1_TECH_SETUP.md`](docs/1_TECH_SETUP.md)\n- **Why are tests confounded?** → [`docs/2_TESTING_STRATEGIES.md`](docs/2_TESTING_STRATEGIES.md)\n- **How do I read results?** → [`docs/5_RUN_REPORTS_EXPLAINED.md`](docs/5_RUN_REPORTS_EXPLAINED.md)\n- **Why does safety matter?** → [`docs/6_USE_CASE_DEADLOCK_SAFETY.md`](docs/6_USE_CASE_DEADLOCK_SAFETY.md)\n- **How do I create a case?** → [`docs/4_HOW_TO_CREATE_USE_CASES.md`](docs/4_HOW_TO_CREATE_USE_CASES.md)\n\n---\n\n## 📄 License\n\n[MIT](LICENSE) © Tzu-Chun Chen and contributors.  \nScribble (vendored in `scribble-java/`) is Apache 2.0 © Imperial College.\n\n## 📖 Citation\n\n```bibtex\n@software{chen2026stjp,\n  author = {Chen, Tzu-Chun},\n  title = {Session-Typed Skills: A static compiler for safe multi-agent interactions},\n  year = {2026},\n  url = {https://github.com/ginaecho/session-typed-skills}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fginaecho%2Fsession-typed-skills","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fginaecho%2Fsession-typed-skills","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fginaecho%2Fsession-typed-skills/lists"}