https://github.com/openclaw/crabpot
Compatibility testbed for OpenClaw community plugins and plugin seams 🦀
https://github.com/openclaw/crabpot
Last synced: about 1 month ago
JSON representation
Compatibility testbed for OpenClaw community plugins and plugin seams 🦀
- Host: GitHub
- URL: https://github.com/openclaw/crabpot
- Owner: openclaw
- License: mit
- Created: 2026-04-25T19:12:03.000Z (about 1 month ago)
- Default Branch: main
- Last Pushed: 2026-04-30T19:57:15.000Z (about 1 month ago)
- Last Synced: 2026-04-30T20:20:34.311Z (about 1 month ago)
- Language: JavaScript
- Homepage:
- Size: 1.77 MB
- Stars: 4
- Watchers: 0
- Forks: 2
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Agents: AGENTS.md
Awesome Lists containing this project
README
# crabpot
Compatibility trap for OpenClaw plugin contracts.
`crabpot` keeps a curated set of real community plugins pinned under `plugins/`
and runs seam-focused compatibility checks against OpenClaw plugin APIs. The goal
is to catch contract drift before external plugin authors do.
## Dashboard
Last dashboard update: Apr 28, 2026, 09:32 UTC
State: PASS
Mode: check
OpenClaw: openclaw/openclaw@main
Run: https://github.com/openclaw/crabpot/actions/runs/25045088692
### Result Grid
| Metric | Result |
| ---------------------- | ---------------------------------------------------- |
| Fixtures | 28 |
| Hard breakages | 0 |
| Warnings | 57 |
| Suggestions | 99 |
| Issues | 156 |
| P0 issues | [🔴 P0 5](reports/crabpot-issues.md#p0-live-issues) |
| P1 issues | [🟠P1 30](reports/crabpot-issues.md#triage-summary) |
| Live issues | 5 total / 5 P0 |
| Compat gaps | 4 |
| Deprecation warnings | 22 |
| Inspector gaps | 101 |
| Upstream metadata | 24 |
| Contract probes | 151 |
| Policy failures | 0 |
| Policy warnings | 3 |
| Ref diff failures | 0 |
| Profile failures | 0 |
| Execution probes | 6 pass / 0 fail / 2 blocked |
| Synthetic probes | 155 ready / 0 blocked / 155 total |
| Cold import | 0 ready / 31 blocked / 31 entrypoints |
| Workspace plan | 31 entrypoints / 20 installs / 8 builds |
| Platform risks | 158 Windows / 49 container |
| Jiti loader candidates | 18 |
| Import loop | p50 153ms / p95 165ms / max RSS 46.3MB / CPU 27ms |
| Runtime profile | p50 649ms / p95 1329ms / max RSS 68.7MB |
### Top Discovered Issues
| Severity | Class | Fixture | Code | Decision | Title |
| -------- | ------------- | ---------------- | ------------------------ | ------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| 🔴 P0 | live-issue | clawmetry | sdk-export-missing | core-compat-adapter | [clawmetry: plugin SDK import aliases are missing from target package exports](reports/crabpot-issues.md#p0-live-issues) |
| 🔴 P0 | live-issue | codex-app-server | sdk-export-missing | core-compat-adapter | [codex-app-server: plugin SDK import aliases are missing from target package exports](reports/crabpot-issues.md#p0-live-issues) |
| 🔴 P0 | live-issue | honcho | sdk-export-missing | core-compat-adapter | [honcho: plugin SDK import aliases are missing from target package exports](reports/crabpot-issues.md#p0-live-issues) |
| 🔴 P0 | live-issue | hyperspell | unknown-hook-name | core-compat-adapter | [hyperspell: fixture uses a hook missing from target OpenClaw](reports/crabpot-issues.md#p0-live-issues) |
| 🔴 P0 | live-issue | yuanbao | sdk-export-missing | core-compat-adapter | [yuanbao: plugin SDK import aliases are missing from target package exports](reports/crabpot-issues.md#p0-live-issues) |
| 🟠P1 | inspector-gap | a2a-gateway | registration-capture-gap | inspector-follow-up | [a2a-gateway: runtime registrations need capture before contract judgment](reports/crabpot-issues.md#inspector-proof-gaps) |
| 🟠P1 | compat-gap | clawmetry | missing-compat-record | core-compat-adapter | [clawmetry: compat-dependent behavior lacks registry coverage](reports/crabpot-issues.md#compat-gaps) |
| 🟠P1 | inspector-gap | clawmetry | registration-capture-gap | inspector-follow-up | [clawmetry: runtime registrations need capture before contract judgment](reports/crabpot-issues.md#inspector-proof-gaps) |
| 🟠P1 | compat-gap | codex-app-server | missing-compat-record | core-compat-adapter | [codex-app-server: compat-dependent behavior lacks registry coverage](reports/crabpot-issues.md#compat-gaps) |
| 🟠P1 | inspector-gap | codex-app-server | registration-capture-gap | inspector-follow-up | [codex-app-server: runtime registrations need capture before contract judgment](reports/crabpot-issues.md#inspector-proof-gaps) |
## What this tests
- plugin manifests and install metadata
- native tool registration and dynamic tool schemas
- channel registration and message delivery seams
- lifecycle hooks such as `gateway_start`, `gateway_stop`, and `before_install`
- agent hooks such as `before_tool_call`, `before_prompt_build`, `llm_input`,
`llm_output`, and `agent_end`
- provider capability registration such as speech/TTS
- plugin-owned services, routes, subprocesses, and async job patterns
## Layout
```text
crabpot/
crabpot.config.json fixture manifest and seam tags
plugins/ external plugin repositories as git submodules
reports/ generated compatibility report artifacts
scripts/ manifest and fixture helpers
test/ repo-level checks
docs/ operating notes and seam matrix
```
## Quick start
```bash
npm test
node scripts/list-fixtures.mjs
node scripts/sync-fixtures.mjs --check
npm run report
npm run contract:capture
npm run contract:synthetic
npm run cold-import
npm run workspace:plan
npm run platform:probes
npm run import:profile
npm run execution:report
npm run profile
npm run contract:coverage
npm run readme:summary
```
To materialize the fixture repos as submodules:
```bash
node scripts/sync-fixtures.mjs --materialize
git submodule update --init --recursive
```
That command mutates `.gitmodules` and `plugins/*`. Commit those changes when
you intentionally pin or update fixture revisions.
## Compatibility report
Start with the dashboard at the top of this README. It is the condensed view of
the generated reports: fixture count, breakages, warnings, issue backlog, probe
coverage, cold-import blockers, workspace execution shape, and runtime profile.
For deeper review, open the reports in this order:
| Need | Command | Primary report |
| --- | --- | --- |
| Main compatibility triage, decision matrix, issue backlog | `npm run report` | `reports/crabpot-report.md` |
| Stable issue list for compat-layer planning | `npm run report` | `reports/crabpot-issues.md` |
| Hooks, registrars, SDK imports, and entrypoints that need capture | `npm run contract:capture` | `reports/crabpot-capture.md` |
| Executable synthetic hook/registration probe plan | `npm run contract:synthetic` | `reports/crabpot-synthetic-probes.md` |
| Why plugin entrypoints cannot be safely cold-imported yet | `npm run cold-import` | `reports/crabpot-cold-import.md` |
| Isolated install/build/capture commands Crabpot would run | `npm run workspace:plan` | `reports/crabpot-workspace-plan.md` |
| Results from opt-in isolated fixture execution | `npm run execution:report` | `reports/crabpot-execution-results.md` |
| Boot time and RSS against the target OpenClaw registry surface | `npm run profile` | `reports/crabpot-runtime-profile.md` |
| README dashboard refresh from all generated JSON reports | `npm run readme:summary` | `README.md` |
Each Markdown report has a matching JSON file beside it for CI, dashboards, and
future inspector tooling. The JSON is the contract; the Markdown is the review
surface.
Use the main compatibility report like this:
| Section | What to do with it |
| --- | --- |
| Hard Breakages | Treat as release-blocking contract drift. |
| Warnings | Review for target OpenClaw compatibility gaps or plugin metadata drift. |
| Suggestions To OpenClaw Compat Layer | Convert into compat-layer work, inspector follow-ups, or upstream plugin requests. |
| Issue Findings | Use stable `CRABPOT-*` ids for tracking and comparison across runs. |
| Contract Probe Backlog | Turn into tests before changing a plugin-facing seam. |
| Decision Matrix | Decide whether the fix belongs in core compat, the future inspector, or the plugin upstream. |
By default, reports target the OpenClaw checkout configured in
`crabpot.config.json`. Point a run at a branch, tag, SHA checkout, or local fork
with `--openclaw`:
```bash
node scripts/generate-report.mjs --openclaw ../openclaw
node scripts/generate-report.mjs --check --openclaw ../openclaw
```
Crabpot does not execute third-party plugin code during default checks. The
workspace plan is dry planning unless you explicitly opt into isolated execution.
Preview a fixture lane first:
```bash
npm run workspace:execute -- --fixture wecom --dry-run
```
Then run isolated execution only when you want install/build/import side effects
inside Crabpot's generated workspace:
```bash
CRABPOT_EXECUTE_ISOLATED=1 npm run workspace:execute -- --fixture wecom
npm run execution:report
```
## Manual OpenClaw ref CI
The `OpenClaw Ref Compatibility` workflow can be run from GitHub Actions with
an OpenClaw branch, tag, or SHA. Set `openclaw_repository` when testing a fork,
and `openclaw_ref` to the exact ref under review.
The default job runs the static contract suite against that checkout and uploads
the generated reports. The optional isolated job runs one fixture lane when
`run_isolated_fixture` is enabled and `fixture` is set, then uploads
`.crabpot/results/` plus the execution summary report.
## Fixture policy
Fixtures should earn their spot by covering a distinct seam. Popularity is a
useful signal, but a small plugin that exercises a rare hook is more valuable
than the fourth web-search wrapper.
The first fixture set intentionally covers channels, dynamic tools, LLM
observation, diagnostics, gateway-owned services, async jobs, provider
capabilities, and security/policy hooks.