{"id":51500102,"url":"https://github.com/ginaecho/skill-achievability-compiler","last_synced_at":"2026-07-07T17:30:32.218Z","repository":{"id":368785505,"uuid":"1286467109","full_name":"ginaecho/skill-achievability-compiler","owner":"ginaecho","description":"a static compiler to validate the achievability of goals of what defined in the skills or agent markdown","archived":false,"fork":false,"pushed_at":"2026-07-02T06:36:54.000Z","size":718,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-02T08:21:04.660Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ginaecho.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-07-01T20:13:29.000Z","updated_at":"2026-07-02T06:15:09.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ginaecho/skill-achievability-compiler","commit_stats":null,"previous_names":["ginaecho/skill-achievability-compiler"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/ginaecho/skill-achievability-compiler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ginaecho%2Fskill-achievability-compiler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ginaecho%2Fskill-achievability-compiler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ginaecho%2Fskill-achievability-compiler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ginaecho%2Fskill-achievability-compiler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ginaecho","download_url":"https://codeload.github.com/ginaecho/skill-achievability-compiler/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ginaecho%2Fskill-achievability-compiler/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35237753,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-07T02:00:07.222Z","response_time":90,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-07T17:30:31.585Z","updated_at":"2026-07-07T17:30:32.191Z","avatar_url":"https://github.com/ginaecho.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# skill-achievability-compiler\n\n**`skillc`** — a static compiler that decides whether the *goal* of an agent\nskill (a `SKILL.md`, agent markdown, or a formal achievability pack) is\nachievable in a given capability context.  It is **sound for refutation**:\nan `IMPOSSIBLE` verdict is a proof (relative to the declared capabilities and\nframe assumption) that no run of the skill can reach its goal.  It is\ndeliberately **incomplete for achievement**: `ACHIEVABLE` means \"structurally\nadmissible\", not \"guaranteed\".\n\nThe soundness core is mechanized in Coq ([`proof/SkillAchievability.v`](proof/SkillAchievability.v),\nzero axioms, audited by [`proof/check_assumptions.v`](proof/check_assumptions.v));\nthe checker decides capability-guarded may-reachability with z3, in\nmilliseconds, with no LLM in the trusted path.  The accompanying paper —\nextended here with full proofs and four implementation-driven strengthenings —\nlives in [`paper/`](paper/) ([PDF](paper/skillachievability.pdf)).\n\n```\n natural-language skill ──► [ front-end compaction ] ──► pack ──► [ checker ] ──► verdict\n      (SKILL.md)              UNTRUSTED                           TRUSTED\n                              deterministic or LLM                sound for refutation\n```\n\n## Install\n\n```bash\npip install -e \".[dev]\"        # installs the `skillc` CLI\n```\n\n## Quick start\n\nCheck a real skill against the runtime that will execute it:\n\n```console\n$ skillc check call-to-book/SKILL.md --profile claude-ai\ncall-to-book: ACHIEVABLE\n\n$ skillc check call-to-book/SKILL.md --profile claude-code\ncall-to-book: IMPOSSIBLE [MISSING_CAPABILITY]\n  protocol invokes undeclared capabilities: ['ask_user_input_v0']\n  missing: ask_user_input_v0 (line 7)\n```\n\nThe same skill, two verdicts: achievability is always judged **relative to a\ncapability context Γ** (an environment *profile* plus the skill's own\n`allowed-tools`/`tools` frontmatter).  A consumer-app skill that asks\nquestions via `ask_user_input_v0` is provably not executable as written under\nClaude Code, which has no such tool.\n\nBatch-scan a skill tree, compile a pack, or run the evaluation corpus:\n\n```console\n$ skillc scan /mnt/skills --profile claude-ai      # 32/32 achievable\n$ skillc compile SKILL.md -o pack.json             # inspect the formal object\n$ skillc check pack.json --json                    # machine-readable verdict\n$ skillc eval                                      # corpus + soundness audit\n$ skillc profiles                                  # claude-ai, claude-code, none\n```\n\nExit codes: `0` achievable, `1` impossible, `2` error, `3` unknown (outside\nthe decidable fragment) — so `skillc check` can\ngate CI for skill repositories.\n\n## What the checker decides\n\nA **pack** declares capabilities (STRIPS pre/effects, numeric assignments,\nconstrained non-determinism), a goal-marked global protocol (`act` / `msg` /\n`choice` / tail-recursive `rec`/`continue` loops / `spawn`), a goal formula,\nthe initial state, and optionally per-role declared behaviours (`skills`).\nThe checker decides **all four premises of the paper's achievability\njudgment** (§5.2):\n\n```\n   Γ ⊇ caps(G)          capability soundness   — no hallucinated tools\n   G ⇓ {T_p}            realizability          — projection defined for every role\n   ∀p. S_p ≤ G↾p        conformance            — declared skills refine their\n                                                  contracts (Gay–Hole subtyping)\n   Γ; G ⊨ ◇goal         liveness               — goal may-reachable (z3)\n   ─────────────────────\n   Γ ⊢ {S_p} : G ▷ ◇goal\n```\n\nProjection implements Proj-Sel / Proj-Brn / Proj-Mrg with the merge `⊓`\n(label-union on external branches); conformance implements Sub-Ext / Sub-Int\ncoinductively.  Refutations name the failing premise:\n\n| reason | failure mode it catches |\n|---|---|\n| `MISSING_CAPABILITY` | hallucinated planning — the protocol invokes a tool that is not granted |\n| `GOAL_UNSAT` | no establisher for a goal conjunct, or a numeric refinement (e.g. *under $500*) unsatisfiable on every run |\n| `BLOCKED_GUARD` | a mandatory action's precondition can never be satisfied (the retry-forever cause) |\n| `NON_PROJECTABLE` | a role must act inside a branch it is never told about and the branches do not merge (unobserved choice → deadlock/handoff freeze) |\n| `NON_CONFORMANT` | a declared role behaviour does not refine its projected contract — the verdict on `G` cannot be transported to it |\n\n**The decidable fragment (Theorems 4–5).** Tail-recursive loops are explored\nwith predicate-state saturation plus numeric widening on the back edge —\nwidening only enlarges the reachable set, so refutation stays sound.  Dynamic\nparticipant spawning (`spawn`) crosses the autonomy boundary\n(Brand–Zafiropulo): the procedure degrades to a semi-decision — it still\nrefutes what survives autonomy and otherwise answers **`UNKNOWN`** (exit\ncode 3) instead of guessing.\n\n**Beyond the original paper** (proved in [`paper/`](paper/), §6.5–§7):\n\n- **Establisher-closure refutation** — a goal conjunct no capability\n  establishes refutes *every* protocol over Γ in one SMT query, spawning\n  included; `GOAL_UNSAT` with a `protocol-independent` certificate then means\n  \"acquire a tool\", not \"fix the plan\".\n- **Observed choice** (`\"observed\": true`) — a choice resolved inside a live\n  conversation (assistant↔user chat, a phone call) is announced by the medium\n  itself; projection treats it as an implicit broadcast (unobserved choice is\n  a ≥3-party *asynchronous* phenomenon).  This eliminated every false\n  refutation of real conversational skills.\n- **Adversarial must-achievability** (`skillc check --adversarial`, choices\n  marked `\"external\": true`) — the goal must survive every environment\n  resolution while the agent's own choices stay existential (AND-OR search);\n  adversarial refutations inherit soundness compositionally from T1.\n- **Counterexample-guided compaction repair** — a `NON_PROJECTABLE`\n  counterexample is fed back to the untrusted compactor for one bounded,\n  structure-only repair round (it may not invent tools or weaken the goal);\n  the verdict always comes from the trusted checker.\n\nTolerance comes from may-reachability (detours allowed), interface slack in\nthe safe direction (a skill may *offer more* receives and *make fewer*\nselections than its contract), and goal-relevant abstraction — extra status\nmessages or beneficial branches never cause a refutation (Coq T2), and\n*adding* capabilities never flips `ACHIEVABLE` to `IMPOSSIBLE` (Coq T3,\n`cap_monotone`).\n\n## Front-ends\n\n1. **Deterministic markdown front-end** (default, no LLM).  Parses\n   frontmatter (`allowed-tools` / `tools`), prose tool declarations\n   (`Tools: a, b, c`), and extracts tool invocations from the prose\n   (\"ask via `ask_user_input_v0`\", \"use `str_replace`\", …).  Unix commands\n   and code symbols route through the profile's shell capability; fenced code\n   blocks are not scanned.  Every extraction is reported with line-number\n   provenance (`skillc compile` prints it to stderr) so the pack is\n   inspectable at one checkpoint.\n2. **Embedded pack**: a fenced block tagged ```` ```skillc-pack ```` inside\n   the SKILL.md is validated and used verbatim — full checker power (guards,\n   budgets, roles, choice) for authors who want precise semantics.\n3. **LLM compaction** (`skillc compile --llm`, needs `ANTHROPIC_API_KEY`):\n   semantic NL→pack distillation.  Untrusted by design — its output passes\n   the same deterministic schema gate and trusted checker, so a hallucinated\n   compaction can only produce a false `ACHIEVABLE` (caught by later layers),\n   never a false `IMPOSSIBLE` about the pack it actually emitted.\n\n## The bundle security pre-pass (`skillc audit`)\n\nBefore the formal pack is trusted, a SkillSpector-like deterministic scanner\nvets the bundle itself — admission control at the compiler's boundary,\ncomplementary to the type discipline:\n\n```console\n$ skillc audit ./my-skill\nmy-skill: ERROR [description-injection] instruction-injection pattern in the description ...\nmy-skill: ERROR [risky-code] pipe-to-shell install (SKILL.md:6)\n```\n\nChecks: manifest consistency (name/description present, name matches the\nbundle directory), metadata poisoning (invisible/bidi Unicode,\ninstruction-injection patterns in the description or hidden in HTML\ncomments), risky code patterns in fenced blocks and bundle scripts\n(pipe-to-shell, decode-and-execute, destructive commands, plaintext HTTP),\nand permission-metadata consistency (prose invocations not covered by\n`allowed-tools`).  Exit 1 on any error-severity finding.  All 32 real public\nbundles pass with zero errors; the planted `poisoned-helper` fixture trips\nevery class.\n\n## Results on real, public skills\n\nValidated against Anthropic's public skills corpus\n([anthropics/skills](https://github.com/anthropics/skills), 32 `SKILL.md`\nfiles mounted at `/mnt/skills`, or fetched with\n`python3 scripts/fetch_skills.py`):\n\n* **32/32 achievable under the `claude-ai` profile** — their home runtime.\n  Zero false refutations on deployed skills (the empirical face of T1).\n* **16/32 refuted under the `claude-code` profile**, each with the exact\n  missing tool named (`ask_user_input_v0`, `read_page`, `upload_file`,\n  `create_file`, `str_replace`, `show_widget`, `search_mcp_registry`, …) and\n  the source line.  Granting the named tools flips every one of them back to\n  achievable (T3 on real data).\n\nFull table: [`docs/REAL_SKILLS_REPORT.md`](docs/REAL_SKILLS_REPORT.md).\n\n**Semantic level** ([`docs/SEMANTIC_VALIDATION.md`](docs/SEMANTIC_VALIDATION.md),\n`scripts/semantic_validation.py`): four representative consumer skills were\nLLM-compacted into semantic packs (goals like *booked ∧ calendar-updated ∧\nuser-informed* with per-step guards), through the schema gate and at most one\nrepair round — **4/4 check ACHIEVABLE** (no false alarms on deployed skills).\nEach pack was then sabotaged with a known ground truth: drop a capability the\nplan invokes, and strip a goal conjunct's establishers — **6/6 mutants\nrefuted**, naming the dropped tool (`MISSING_CAPABILITY`) or the dead conjunct\n(`GOAL_UNSAT`, protocol-independent certificate) every time.  Deterministic\nmutation testing over all 32 skills works in both directions: removing an\ninvoked tool from the profile flips the verdict and names exactly that tool;\ngranting the frontier back flips it to achievable.\n\nOn the 15-spec ground-truth corpus (`skillc eval`): **FN = 0** (no achievable\ngoal ever refuted — T1) and the only false `ACHIEVABLE`s are the two planted\n`SPURIOUS` cases (payload faithfulness / intent fidelity), i.e. exactly the\nresidues the compiler openly defers to runtime monitoring and human review —\nnever a structural failure it should have caught.\n\n## Tests\n\n```bash\npython3 -m pytest                          # 236 tests\nSKILLC_SKILLS_DIR=./real-skills pytest tests/test_real_skills.py   # real corpus\nSKILLC_LIVE_LLM=1 pytest tests/test_llm_frontend.py               # live LLM (opt-in)\ncoqc proof/SkillAchievability.v \u0026\u0026 coqc proof/check_assumptions.v  # the proof\n```\n\nThe suite covers the formula language, the schema gate, every refutation\nreason, projection/merge/subtyping, tail recursion and the autonomy boundary\n(loops saturate, `spawn` degrades to `UNKNOWN`, refutation survives\nautonomy), conformance (Sub-Ext/Sub-Int both directions), the corpus\nconfusion matrix (reproduced exactly: TP=6 FN=0 FP=2 TN=7) plus a 6-spec\nextended corpus for the fragment boundary, the markdown front-end\n(extraction, classification, profiles, embedded packs), the bundle audit\n(poisoned fixture trips every class; real bundles are clean), the CLI, and —\nwhen a corpus is present — every real public skill under multiple profiles,\nincluding the monotone-widening property.\n\n## Layout\n\n```\nsrc/skillc/            the compiler package\n  checker.py             trusted core: the four-premise judgment over z3\n                         may-reachability, loops + widening, UNKNOWN boundary\n  session.py             projection (Proj-Sel/Brn/Mrg), merge, Gay-Hole subtyping\n  pack.py                pack model + deterministic schema gate\n  formula.py             guard/goal mini-language\n  profiles.py            capability contexts (claude-ai, claude-code, none)\n  audit.py               SkillSpector-like bundle security pre-pass\n  frontend/markdown.py   deterministic SKILL.md -\u003e pack compaction\n  frontend/llm.py        optional LLM compaction (untrusted, env-gated)\n  evaluate.py            corpus evaluation + soundness/incompleteness audit\n  cli.py                 skillc compile | check | scan | audit | eval | profiles\n  data/                  built-in profiles + evaluation corpora\npaper/                 the paper (LaTeX + built PDF): full proofs +\n                       implementation-driven extensions\nproof/                 the theorem checkers for the paper's claims (Coq 8.18,\n                       zero axioms) -- SkillAchievability.v certifies T1/T2/T3;\n                       DirectTyping.v the direct-typing head-move safety;\n                       DirectTypingSR.v subject reduction + session fidelity\n                       (communication interleaving); the compiler itself is the\n                       Python package above\ncorpus/build_corpus.py 15 headline specs + 6 fragment/conformance specs\ndocs/                  compaction prompt + real-skill scan report\nexamples/              embedded-pack skills: retry loop, conformance-checked team\nscripts/               fetch_skills.py, make_report.py\ntests/                 the test suite (pytest)\n```\n\n## Honest limitations\n\nEverything is proved about the *declared* capabilities and protocol; if the\nprose lies, the checker verifies a fiction (honest declaration is a runtime\nobligation — the `skillc audit` pre-pass narrows, but does not close, that\ngap).  The deterministic front-end is a conservative heuristic — its\nextraction is inspectable and a misextraction only makes the checker judge a\ndifferent pack, but it does not understand semantics (use the embedded-pack\nescape hatch or `--llm` for that).  The merge `⊓` implements label-union on\nexternal branches and structural recursion on equal prefixes — a sound core\nof, not the complete, MPST merge lattice; loop widening havocs *all* numeric\nstate at the back edge (coarser than needed, always in the sound direction).\nDynamic subagent spawning is outside the decidable fragment and yields\n`UNKNOWN`, never a guess.\n\n## How to cite\n\nThe skill-achievability-compiler is an open-source project. If you use it in academic work, research papers, or other publications, please cite it using one of the formats below:\n\n**BibTeX:**\n```bibtex\n@software{skillc2026,\n  title = {skill-achievability-compiler: A Static Compiler for Goal Achievability Verification in LLM-Synthesized Agent Skills},\n  author = {Tcchen, Gina},\n  year = {2026},\n  url = {https://github.com/ginaecho/skill-achievability-compiler},\n  license = {MIT}\n}\n```\n\n**APA:**\n```\nTcchen, G. (2026). skill-achievability-compiler: A static compiler for goal achievability verification in LLM-synthesized agent skills. Retrieved from https://github.com/ginaecho/skill-achievability-compiler\n```\n\n**Chicago:**\n```\nTcchen, Gina. \"skill-achievability-compiler: A Static Compiler for Goal Achievability Verification in LLM-Synthesized Agent Skills.\" GitHub. Accessed [date]. https://github.com/ginaecho/skill-achievability-compiler.\n```\n\nGitHub will also recognize the [`CITATION.cff`](CITATION.cff) file in this repository, which provides machine-readable citation metadata.\n\n## License\n\nThis project is licensed under the **MIT License** — see the [`LICENSE`](LICENSE) file for details.\n\nCopyright © 2026 Gina Tcchen and contributors.\n\nThe MIT License permits free use, modification, distribution, and sublicensing of the software, provided that the copyright notice and license text are retained. This means:\n\n✓ You **can** use this software for any purpose (commercial, private, etc.)  \n✓ You **can** modify and distribute the software  \n✓ You **can** include it in proprietary applications  \n✓ You **must** include the license and copyright notice  \n✓ The software is provided **as-is**, with no warranty\n\nFor full legal terms, see the [LICENSE](LICENSE) file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fginaecho%2Fskill-achievability-compiler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fginaecho%2Fskill-achievability-compiler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fginaecho%2Fskill-achievability-compiler/lists"}