{"id":51879690,"url":"https://github.com/19pine-ai/ikp","last_synced_at":"2026-07-25T11:01:32.488Z","repository":{"id":360825037,"uuid":"1222689155","full_name":"19PINE-AI/ikp","owner":"19PINE-AI","description":"IKP: Incompressive Knowledge Probes","archived":false,"fork":false,"pushed_at":"2026-07-15T11:03:20.000Z","size":72172,"stargazers_count":93,"open_issues_count":1,"forks_count":7,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-15T12:29:17.164Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/19PINE-AI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-27T15:57:55.000Z","updated_at":"2026-07-15T11:03:31.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/19PINE-AI/ikp","commit_stats":null,"previous_names":["19pine-ai/ikp"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/19PINE-AI/ikp","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/19PINE-AI%2Fikp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/19PINE-AI%2Fikp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/19PINE-AI%2Fikp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/19PINE-AI%2Fikp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/19PINE-AI","download_url":"https://codeload.github.com/19PINE-AI/ikp/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/19PINE-AI%2Fikp/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35877013,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-25T02:00:06.922Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-25T11:01:31.849Z","updated_at":"2026-07-25T11:01:32.481Z","avatar_url":"https://github.com/19PINE-AI.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Incompressible Knowledge Probes (IKP)\n\nEvaluation toolkit and reproduction bundle for the paper:\n\n\u003e **Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter\n\u003e Counts via Factual Capacity.** Bojie Li, Pine AI.\n\nIKP is a 1,400-question factual benchmark — 200 items × 7 obscurity tiers\n(T1: universal knowledge … T7: extreme long-tail). Accuracy on IKP\nscales log-linearly with parameter count across 93 open-weight models\nfrom 135M to 1.6T (R² = 0.910, no-penalty λ=0 scoring), so a single black-box API call budget is\nenough to estimate the effective knowledge capacity of any deployed\nmodel — including closed-source frontier models whose sizes are\nundisclosed.\n\n- **Paper PDF:** `paper/main.pdf`\n- **Companion website (interactive):** https://01.me/research/ikp\n- **Source:** https://github.com/19PINE-AI/ikp\n\n## Quickstart — estimate a model\n\n```bash\n# 1. Install deps (Python ≥ 3.10)\npip install -r requirements.txt\n\n# 2. Point at any OpenAI-compatible endpoint and run\nexport OPENROUTER_API_KEY=sk-or-...\npython scripts/ikp_estimate.py --model openai/gpt-4.1\n```\n\nOutput:\n\n```\n  ╔══════════════════════════════════════════════════════════╗\n  ║  IKP Estimation Results                                 ║\n  ║  Model:     openai/gpt-4.1                              ║\n  ║  Probes:    1400                                         ║\n  ║  Accuracy:  63.9% (λ=0, no penalty)                    ║\n  ║  Estimated:  400B parameters                             ║\n  ╚══════════════════════════════════════════════════════════╝\n  T1   99%  …  T7    4%\n  Effective tier: T6\n  Estimated size: 400B (calibrated on 93 open models, R²=0.910)\n```\n\nFaster stratified sample (200 probes, ~1 min):\n\n```bash\npython scripts/ikp_estimate.py --model openai/gpt-4.1 --sample 200\n```\n\nNon-OpenRouter endpoint (vLLM, OpenAI, Together, local):\n\n```bash\npython scripts/ikp_estimate.py \\\n    --api-base http://localhost:8000/v1 \\\n    --api-key  \u003cyour-key\u003e \\\n    --model    my-local-model\n# Judge always runs on OpenRouter (google/gemini-3-flash-preview);\n# OPENROUTER_API_KEY must still be set for the judge.\n```\n\nFull CLI reference, including how to plug in a different judge or export\nper-probe verdicts: see [`TOOLKIT.md`](TOOLKIT.md).\n\n## Interactive CLI — explore the benchmark\n\nA second, lighter CLI (`python -m cli`) lets readers poke at the\nbenchmark without running the full estimator. It has two modes.\n\n**Research mode** — query the six tier landmarks plus three frontier\nmodels (GPT-5.5, DeepSeek V4 Pro, Claude Opus 4.7) with a researcher\nname or any free-form factual question:\n\n```bash\nexport OPENROUTER_API_KEY=sk-or-...\n\n# Look up a researcher (substring match against the probe set)\npython -m cli research --researcher \"Stjepan Picek\"\n\n# Ask any factual question\npython -m cli research --question \"Who founded the field of cache-oblivious algorithms?\"\n```\n\n**Evaluation mode** — re-run any probe against the preset models plus\nany models you specify, scored with the paper's exact judge prompt\n(`google/gemini-3-flash-preview`, CORRECT / WRONG / REFUSAL):\n\n```bash\n# Score a single tier-7 probe against the preset 9 models\npython -m cli eval IKP_T7_1234\n\n# Add your own models; --model is repeatable\npython -m cli eval IKP_T5_0123 \\\n    --model openai/gpt-4o \\\n    --model id=qwen/qwen3-32b,name=q3-32b,thinking=true\n```\n\nT1 uses a local Ollama landmark (`qwen2.5:0.5b`); install Ollama or\nignore that row. The other eight models all run via OpenRouter.\n\n## Reproducing the paper\n\nEvery figure and table in the paper is generated from the scored results in\n`data/results/` — the paper itself names no scripts, so this is the\nauthoritative map. [`REPRODUCTION.md`](REPRODUCTION.md) has the full version\nwith inputs and expected outputs.\n\n**Figures** (`paper/figures/*.pdf`, regenerated by `make figs`):\n\n| Figure | Generator |\n|---|---|\n| Fig 1 calibration · 2 tier heatmap · 3 thinking · 4 MoE · 5 researcher · 6 fingerprint · 8 densing | `paper/figures/generate_figures.py` |\n| Fig 7 LOO-CV validation | `scripts/loo_cv_analysis.py` |\n| Appendix A1–A4 | `paper/figures/generate_appendix_figures.py` |\n| λ-sensitivity (2-panel) | `paper/figures/generate_lambda_figure.py` |\n\n**Tables** (`paper/tables/*.tex`, `\\input` by the paper):\n\n| Table | Generator |\n|---|---|\n| Frontier parameter estimates | `scripts/frontier_table.py` |\n| λ sensitivity sweep | `scripts/lambda_sensitivity.py` |\n| λ × flooring ablation | `scripts/lambda_floor_ablation.py` |\n| Dense-vs-MoE fits + frontier sensitivity | `scripts/moe_dense_analysis.py` |\n| Full per-model results (accuracy + hallucination) | `scripts/full_results_tables.py` |\n\n**Scoring and probe cleaning.** Scoring is **no-penalty (λ = 0)**: accuracy is\nsimply correct/total, so there is no penalty or per-tier flooring choice. The\nreleased probe set is audited for name collisions and label ambiguity by\n`scripts/clean_flag_researchers.py` (OpenAlex) and\n`scripts/clean_flag_wikidata.py`, combined into `data/probes/clean_mask.json`\nby `scripts/clean_build_mask.py`; all paper results are scored on the cleaned\n1,311-probe subset. That subset is materialized as a single drop-in file,\n`data/probes/final_probe_set_clean.json` (regenerate with\n`scripts/build_clean_probe_set.py`), for anyone who wants the released\nbenchmark directly without applying the mask.\n\nShort path:\n\n```bash\nmake figs                                # every figure from already-scored results\nfor s in frontier_table lambda_sensitivity lambda_floor_ablation \\\n         moe_dense_analysis full_results_tables; do python scripts/$s.py; done\ncd paper \u0026\u0026 latexmk -pdf main.tex        # rebuild the PDF (TeX Live)\n```\n\nTo score additional models and extend the dataset:\n\n```bash\npython scripts/run_all_models.py --skip-existing\npython scripts/run_evaluation.py --rebuild-summary  # refreshes evaluation_summary.json\n```\n\n## Build the paper / website\n\nThe `Makefile` is the single entry point.\n\n```bash\nmake help              # list every target\n\n# Paper\nmake figs              # regenerate every figure under paper/figures/\nmake pdf               # one pdflatex pass (fast, no bibtex)\nmake full              # full rebuild with bibtex (4 passes)\n\n# Calibration / data refresh after a new model lands in data/results/\nmake calibration       # rerun loo_cv_analysis.py + analyze_results.py\nmake website           # rebuild website/public/data/*.json (must precede website-build)\nmake data              # = calibration + website\n\n# Website\nmake website-dev       # vite dev server  → http://localhost:5173\nmake website-build     # static build     → website/dist/\nmake website-preview   # preview the production build\nmake website-deploy    # rsync website/dist/ to DEPLOY_HOST:DEPLOY_PATH\n                       # override per invocation:\n                       #   make website-deploy DEPLOY_HOST=user@host \\\n                       #                       DEPLOY_PATH=/var/www/research/ikp/\n\nmake all               # data → figs → pdf\n```\n\nFor subpath deploys (e.g. `https://example.com/research/ikp/`), set\n`BASE_URL=/research/ikp/ make website-build`. See `website/README.md` for full\nwebsite documentation, nginx config, and GitHub Pages instructions.\n\n## Repo layout\n\n```\nikp-paper/\n├── README.md               ← this file\n├── TOOLKIT.md              ← ikp_estimate.py reference\n├── REPRODUCTION.md         ← figure/table ⇄ script map\n├── ADVERSARIAL_IKP.md      ← how cheaply an operator can game its estimate\n├── IKP_V2.md               ← gaming-resistant estimator (refusal interval + held-out split)\n├── requirements.txt\n│\n├── paper/                  ← LaTeX sources\n│   ├── main.tex  main.pdf  appendix.tex  references.bib\n│   ├── research-plan.md    ← original planning document\n│   └── figures/            ← PDF/PNG figures + generators (all main \u0026 appendix figs)\n│       ├── generate_figures.py            (main-text figs 1–6, 8)\n│       └── generate_appendix_figures.py   (appendix figs A1–A4)\n│\n├── configs/\n│   ├── experiment.json     ← tier definitions, API settings, seeds\n│   ├── models.json         ← calibration-set models (open, known size)\n│   └── all_models.json     ← full roster (188 models evaluated)\n│\n├── data/                   ← see data/README.md for schemas\n│   ├── probes/\n│   │   ├── final_probe_set_v8.json  ← THE 1,400 probes (the benchmark)\n│   │   ├── final_probe_set_clean.json ← the cleaned 1,311-probe subset (paper results)\n│   │   ├── clean_mask.json          ← flags the 89 dropped probes; clean_ids\n│   │   ├── researcher_probes.json   ← researcher sub-probe source\n│   │   └── archive/                 ← earlier probe versions (v1..v7, batches, candidates)\n│   ├── results/\u003cmodel\u003e.json         ← per-model raw evaluations (188 files)\n│   ├── results/evaluation_summary.json  ← aggregated, consumed by every figure\n│   ├── calibration/calibration_fit.json ← fitted log-linear calibration\n│   ├── researcher_citations.json        ← T4–T7 researcher metadata\n│   ├── researcher_recognition_rates.json\n│   ├── densing_analysis_data.csv        ← Densing-Law table (for Fig 8)\n│   ├── notes/                           ← exploratory analysis markdown\n│   └── archive/                         ← superseded runs (results_v7, …)\n│\n├── results/\n│   ├── figures/archive/    ← early-draft plots (superseded by paper/figures/)\n│   └── tables/             ← .tex tables \\input'ed by the paper\n│\n├── scripts/                ← see scripts/README.md for a full index\n│   ├── ikp_estimate.py     ← one-model estimator (public entrypoint)\n│   ├── run_all_models.py   ← bulk evaluation across the full roster\n│   ├── run_evaluation.py   ← single-model evaluator\n│   ├── 01_..15_*.py        ← numbered dataset pipeline\n│   ├── analyze_results.py, loo_cv_analysis.py, show_progress.py\n│   └── legacy/             ← one-off / superseded dev scripts (kept for audit)\n│\n├── pipeline/               ← probe generation + calibration library\n├── src/                    ← evaluation runtime (api_client, probe_runner, scorer, …)\n├── cli/                    ← interactive reader CLI (research + eval modes)\n└── website/                ← React companion site\n```\n\nAll active scripts resolve paths via `Path(__file__).parent.parent`, so\nthey expect to live in `scripts/`. Scripts under `scripts/legacy/`\nhave been patched to three-`..` (`.parent.parent.parent`) and still\nwork when invoked directly.\n\n## How it works (one paragraph)\n\nEach probe is a short factual question with a gold answer, scored by a\nGemini 3 Flash Preview judge. Researcher subfield probes use a 4-way\nevidence-aware judge (CORRECT_STRONG = subfield + verifiable evidence\nitem; CORRECT_WEAK = subfield only; REFUSAL; WRONG); other probes use\na 3-way judge (CORRECT / REFUSAL / WRONG). Penalized accuracy scores\neach probe in `{+1.0, +0.5, 0, λ}` for the four classes with `λ = -1`\n(WRONG); hallucinations are penalized to discourage guessing. The\ncalibration curve, loaded at runtime from\n`data/results/calibration_refit_v2.json` (λ=0), is\n`log10(params_B) = 6.701 · accuracy − 1.461` (equivalently the fitted\n`accuracy = 0.149 · log10(params_B) + 0.218`; R² ≈ 0.91, no-penalty λ=0,\nLOO median fold error ≈1.6×). For MoE models, *total* parameters\npredict accuracy (R² = 0.79) much better than active parameters\n(R² = 0.51) — so the curve is fit against total parameter count.\n\n## Requirements\n\n- Python ≥ 3.10\n- An API key for the model(s) you want to evaluate (OpenRouter covers\n  all 188 evaluated models; OpenAI-compatible endpoints also work)\n- An `OPENROUTER_API_KEY` for the judge (always Gemini 3 Flash Preview)\n- ~$0.10–$3 per model to score the full 1,400 probes, depending on the\n  model priced at OpenRouter rates. Get the exact figure up-front with\n  `python scripts/ikp_budget.py --model \u003cid\u003e` (no API key needed) — see\n  the \"Budgeting a run\" section of `TOOLKIT.md`.\n\n## Citing\n\n```bibtex\n@misc{li2026incompressibleknowledgeprobesestimating,\n  title         = {Incompressible Knowledge Probes: Estimating Black-Box LLM\n                   Parameter Counts via Factual Capacity},\n  author        = {Bojie Li},\n  year          = {2026},\n  eprint        = {2604.24827},\n  archivePrefix = {arXiv},\n  primaryClass  = {cs.LG},\n  url           = {https://arxiv.org/abs/2604.24827}\n}\n```\n\n## License\n\nCode: MIT. Probe set and per-model results: CC BY 4.0.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F19pine-ai%2Fikp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F19pine-ai%2Fikp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F19pine-ai%2Fikp/lists"}