{"id":51013656,"url":"https://github.com/ibidathoillah/openai-compatible-tester-cli","last_synced_at":"2026-06-21T07:01:51.993Z","repository":{"id":361942730,"uuid":"1256510937","full_name":"ibidathoillah/openai-compatible-tester-cli","owner":"ibidathoillah","description":"Rust CLI to test whether OpenAI-compatible APIs are truly compatible across chat, streaming, tools, structured outputs, embeddings, reports, and CI.","archived":false,"fork":false,"pushed_at":"2026-06-01T22:21:42.000Z","size":69,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-02T00:15:04.029Z","etag":null,"topics":["api-testing","cli","compatibility-testing","developer-tools","llm","llmops","openai","openai-compatible","rust","testing"],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ibidathoillah.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-01T21:06:16.000Z","updated_at":"2026-06-01T22:21:45.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ibidathoillah/openai-compatible-tester-cli","commit_stats":null,"previous_names":["ibidathoillah/openai-compatible-tester-cli"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/ibidathoillah/openai-compatible-tester-cli","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ibidathoillah%2Fopenai-compatible-tester-cli","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ibidathoillah%2Fopenai-compatible-tester-cli/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ibidathoillah%2Fopenai-compatible-tester-cli/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ibidathoillah%2Fopenai-compatible-tester-cli/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ibidathoillah","download_url":"https://codeload.github.com/ibidathoillah/openai-compatible-tester-cli/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ibidathoillah%2Fopenai-compatible-tester-cli/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34597337,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-21T02:00:05.568Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api-testing","cli","compatibility-testing","developer-tools","llm","llmops","openai","openai-compatible","rust","testing"],"created_at":"2026-06-21T07:01:50.994Z","updated_at":"2026-06-21T07:01:51.987Z","avatar_url":"https://github.com/ibidathoillah.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# openai-compatible-tester-cli\n\n**A fast Rust CLI for testing whether an \"OpenAI-compatible\" API is actually compatible.**\n\nMany providers, gateways, proxies, and local inference servers claim OpenAI compatibility. In practice, compatibility often breaks around streaming, tool calling, structured outputs, usage metadata, error shapes, SDK parsing, and ignored parameters.\n\n```txt\ncurl tells you the API is alive.\noctest tells you whether it behaves like an OpenAI-compatible API.\n```\n\n`openai-compatible-tester-cli` ships the `octest` binary: a CI-friendly compatibility tester for developers building LLM gateways, provider APIs, local model servers, and agent infrastructure.\n\n## Why\n\nRaw HTTP checks are not enough. The dangerous failures are usually silent:\n\n- `tool_choice` is accepted but ignored.\n- `response_format` returns JSON, but not the requested schema.\n- Streaming works, but not as valid SSE.\n- `usage` is missing, misplaced, or inconsistent.\n- Invalid models silently fall back to another model.\n- A request works with `curl`, then fails in an SDK or agent framework.\n\n`octest` is built to catch those compatibility gaps and turn them into repeatable reports.\n\n## Current MVP\n\nImplemented now:\n\n| Area | Coverage |\n|---|---|\n| Models | `GET /models`, `GET /models/{model}` |\n| Chat | Basic, system message, multi-turn, common parameters |\n| Streaming | SSE parsing, `[DONE]`, first-token timing, stream usage |\n| Agent | Tool calling, tool result flow |\n| Schema | JSON mode, strict structured output checks |\n| Embeddings | Single and batch input |\n| Errors | Invalid model, invalid JSON, readable error shape |\n| Reports | Terminal, JSON, Markdown |\n| Safety | API key redaction by default |\n| Local testing | Built-in mock OpenAI-compatible server |\n\nPlanned next: SDK compatibility, problem scan/conformance mode, provider dialects, ignored-parameter detection, performance profile, cost/billing guardrails, regression diff, JUnit/HTML reports, testpacks, and broader endpoint coverage.\n\n## Install\n\nFrom this checkout:\n\n```bash\ncargo install --path .\n```\n\nThen:\n\n```bash\noctest --help\n```\n\n## Quick Local Demo\n\nStart the built-in mock server:\n\n```bash\noctest mock-server --port 8080\n```\n\nRun a quick compatibility scan:\n\n```bash\noctest quick \\\n  --base-url http://localhost:8080/v1 \\\n  --model mock-chat \\\n  --no-auth\n```\n\n## Test a Provider\n\nCore compatibility:\n\n```bash\noctest run \\\n  --base-url https://api.example.com/v1 \\\n  --api-key-env PROVIDER_API_KEY \\\n  --model gpt-compatible \\\n  --profile core \\\n  --output report.json\n```\n\nAgent compatibility:\n\n```bash\noctest run \\\n  --base-url https://api.example.com/v1 \\\n  --api-key-env PROVIDER_API_KEY \\\n  --model gpt-compatible \\\n  --profile agent \\\n  --format markdown \\\n  --output COMPATIBILITY.md\n```\n\nEmbeddings:\n\n```bash\noctest embeddings \\\n  --base-url https://api.example.com/v1 \\\n  --api-key-env PROVIDER_API_KEY \\\n  --embedding-model text-embedding-compatible\n```\n\n## Config File\n\nCreate a starter config:\n\n```bash\noctest init provider.yaml\n```\n\nRun from config:\n\n```bash\noctest run -c provider.yaml\n```\n\nCLI flags override YAML values. API keys are read from `--api-key`, then `--api-key-env`, and are redacted from output and reports.\n\n## Reports\n\nSupported MVP report formats:\n\n- `terminal`\n- `json`\n- `markdown`\n\nConvert a saved JSON report:\n\n```bash\noctest report report.json --format markdown --output COMPATIBILITY.md\n```\n\nExample JSON report shape:\n\n```json\n{\n  \"score\": {\n    \"overall\": 100,\n    \"max\": 100,\n    \"grade\": \"full_compatible\"\n  },\n  \"features\": {\n    \"chat_completions\": \"passed\",\n    \"streaming\": \"passed\",\n    \"tool_calling\": \"passed\",\n    \"structured_outputs\": \"passed\"\n  }\n}\n```\n\n## CI Behavior\n\nExit codes are designed for automation:\n\n| Code | Meaning |\n|---:|---|\n| 0 | Required tests passed |\n| 1 | At least one required test failed |\n| 2 | Invalid config or client setup |\n| 5 | Internal CLI error |\n| 6 | Score below `--min-score` |\n\nExample:\n\n```bash\noctest run \\\n  --base-url \"$OPENAI_COMPAT_BASE_URL\" \\\n  --api-key \"$OPENAI_COMPAT_API_KEY\" \\\n  --model \"$OPENAI_COMPAT_MODEL\" \\\n  --profile core \\\n  --format json \\\n  --output report.json \\\n  --min-score 80\n```\n\nSee [examples/github-action.yaml](examples/github-action.yaml) for a GitHub Actions starter workflow.\n\n## Roadmap\n\nThe next product milestone is production readiness, not just raw HTTP coverage:\n\n- **SDK compatibility mode** for `openai-python`, `openai-node`, Vercel AI SDK, LiteLLM, and LangChain.\n- **Problem-driven conformance scan** for fake OpenAI-compatible behavior: silent fallback, ignored tools/schema/token limits, non-standard usage, and SDK parse failures.\n- **Provider dialects and adapters** for OpenAI strict, OpenAI legacy, Azure OpenAI, reasoning models, and local LLM servers.\n- **Ignored-parameter detection** for `max_tokens`, `stop`, forced `tool_choice`, JSON mode, strict schemas, and `n`.\n- **Performance profile** with p50/p95/p99 latency, TTFT, tokens/sec, small-concurrency stability, warmup requests, thresholds, and perf diff.\n- **Cost and billing guardrails** with pricing YAML, dry-run estimates, budget blocking, usage consistency checks, billing metadata checks, and cost regression.\n- **Production readiness checks** for rate limits, token accounting, protocol behavior, security/redaction, SLOs, regression snapshots, debug replay, and resource cleanup.\n\nRead [PRD.md](PRD.md) for the full product plan and endpoint coverage registry.\n\n## Development\n\nRun tests:\n\n```bash\ncargo test\n```\n\nFormat:\n\n```bash\ncargo fmt --all\n```\n\nThe current codebase includes focused unit tests for config merging, scoring, report rendering, registry selection, redaction, JSON helpers, and client parsing utilities.\n\n## Security\n\n`octest` treats secrets as sensitive by default:\n\n- API keys are read from flags or environment variables.\n- Authorization values are redacted from output.\n- Reports are designed to be safe to commit.\n- Raw debug dumps are not enabled by default.\n\nDo not run costly or destructive endpoint families without explicit flags when those features are added.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fibidathoillah%2Fopenai-compatible-tester-cli","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fibidathoillah%2Fopenai-compatible-tester-cli","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fibidathoillah%2Fopenai-compatible-tester-cli/lists"}