Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/19pine-ai/ikp

IKP: Incompressive Knowledge Probes
https://github.com/19pine-ai/ikp

Last synced: 16 days ago
JSON representation

IKP: Incompressive Knowledge Probes

Awesome Lists containing this project

README 

          
# Incompressible Knowledge Probes (IKP)



Evaluation toolkit and reproduction bundle for the paper:



> **Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter

> Counts via Factual Capacity.** Bojie Li, Pine AI.



IKP is a 1,400-question factual benchmark — 200 items × 7 obscurity tiers

(T1: universal knowledge … T7: extreme long-tail). Accuracy on IKP

scales log-linearly with parameter count across 93 open-weight models

from 135M to 1.6T (R² = 0.910, no-penalty λ=0 scoring), so a single black-box API call budget is

enough to estimate the effective knowledge capacity of any deployed

model — including closed-source frontier models whose sizes are

undisclosed.



- **Paper PDF:** `paper/main.pdf`

- **Companion website (interactive):** https://01.me/research/ikp

- **Source:** https://github.com/19PINE-AI/ikp



## Quickstart — estimate a model



```bash

# 1. Install deps (Python ≥ 3.10)

pip install -r requirements.txt



# 2. Point at any OpenAI-compatible endpoint and run

export OPENROUTER_API_KEY=sk-or-...

python scripts/ikp_estimate.py --model openai/gpt-4.1

```



Output:



```

  ╔══════════════════════════════════════════════════════════╗

  ║  IKP Estimation Results                                 ║

  ║  Model:     openai/gpt-4.1                              ║

  ║  Probes:    1400                                         ║

  ║  Accuracy:  63.9% (λ=0, no penalty)                    ║

  ║  Estimated:  400B parameters                             ║

  ╚══════════════════════════════════════════════════════════╝

  T1   99%  …  T7    4%

  Effective tier: T6

  Estimated size: 400B (calibrated on 93 open models, R²=0.910)

```



Faster stratified sample (200 probes, ~1 min):



```bash

python scripts/ikp_estimate.py --model openai/gpt-4.1 --sample 200

```



Non-OpenRouter endpoint (vLLM, OpenAI, Together, local):



```bash

python scripts/ikp_estimate.py \

    --api-base http://localhost:8000/v1 \

    --api-key   \

    --model    my-local-model

# Judge always runs on OpenRouter (google/gemini-3-flash-preview);

# OPENROUTER_API_KEY must still be set for the judge.

```



Full CLI reference, including how to plug in a different judge or export

per-probe verdicts: see [`TOOLKIT.md`](TOOLKIT.md).



## Interactive CLI — explore the benchmark



A second, lighter CLI (`python -m cli`) lets readers poke at the

benchmark without running the full estimator. It has two modes.



**Research mode** — query the six tier landmarks plus three frontier

models (GPT-5.5, DeepSeek V4 Pro, Claude Opus 4.7) with a researcher

name or any free-form factual question:



```bash

export OPENROUTER_API_KEY=sk-or-...



# Look up a researcher (substring match against the probe set)

python -m cli research --researcher "Stjepan Picek"



# Ask any factual question

python -m cli research --question "Who founded the field of cache-oblivious algorithms?"

```



**Evaluation mode** — re-run any probe against the preset models plus

any models you specify, scored with the paper's exact judge prompt

(`google/gemini-3-flash-preview`, CORRECT / WRONG / REFUSAL):



```bash

# Score a single tier-7 probe against the preset 9 models

python -m cli eval IKP_T7_1234



# Add your own models; --model is repeatable

python -m cli eval IKP_T5_0123 \

    --model openai/gpt-4o \

    --model id=qwen/qwen3-32b,name=q3-32b,thinking=true

```



T1 uses a local Ollama landmark (`qwen2.5:0.5b`); install Ollama or

ignore that row. The other eight models all run via OpenRouter.



## Reproducing the paper



Every figure and table in the paper is generated from the scored results in

`data/results/` — the paper itself names no scripts, so this is the

authoritative map. [`REPRODUCTION.md`](REPRODUCTION.md) has the full version

with inputs and expected outputs.



**Figures** (`paper/figures/*.pdf`, regenerated by `make figs`):



| Figure | Generator |

|---|---|

| Fig 1 calibration · 2 tier heatmap · 3 thinking · 4 MoE · 5 researcher · 6 fingerprint · 8 densing | `paper/figures/generate_figures.py` |

| Fig 7 LOO-CV validation | `scripts/loo_cv_analysis.py` |

| Appendix A1–A4 | `paper/figures/generate_appendix_figures.py` |

| λ-sensitivity (2-panel) | `paper/figures/generate_lambda_figure.py` |



**Tables** (`paper/tables/*.tex`, `\input` by the paper):



| Table | Generator |

|---|---|

| Frontier parameter estimates | `scripts/frontier_table.py` |

| λ sensitivity sweep | `scripts/lambda_sensitivity.py` |

| λ × flooring ablation | `scripts/lambda_floor_ablation.py` |

| Dense-vs-MoE fits + frontier sensitivity | `scripts/moe_dense_analysis.py` |

| Full per-model results (accuracy + hallucination) | `scripts/full_results_tables.py` |



**Scoring and probe cleaning.** Scoring is **no-penalty (λ = 0)**: accuracy is

simply correct/total, so there is no penalty or per-tier flooring choice. The

released probe set is audited for name collisions and label ambiguity by

`scripts/clean_flag_researchers.py` (OpenAlex) and

`scripts/clean_flag_wikidata.py`, combined into `data/probes/clean_mask.json`

by `scripts/clean_build_mask.py`; all paper results are scored on the cleaned

1,311-probe subset. That subset is materialized as a single drop-in file,

`data/probes/final_probe_set_clean.json` (regenerate with

`scripts/build_clean_probe_set.py`), for anyone who wants the released

benchmark directly without applying the mask.



Short path:



```bash

make figs                                # every figure from already-scored results

for s in frontier_table lambda_sensitivity lambda_floor_ablation \

         moe_dense_analysis full_results_tables; do python scripts/$s.py; done

cd paper && latexmk -pdf main.tex        # rebuild the PDF (TeX Live)

```



To score additional models and extend the dataset:



```bash

python scripts/run_all_models.py --skip-existing

python scripts/run_evaluation.py --rebuild-summary  # refreshes evaluation_summary.json

```



## Build the paper / website



The `Makefile` is the single entry point.



```bash

make help              # list every target



# Paper

make figs              # regenerate every figure under paper/figures/

make pdf               # one pdflatex pass (fast, no bibtex)

make full              # full rebuild with bibtex (4 passes)



# Calibration / data refresh after a new model lands in data/results/

make calibration       # rerun loo_cv_analysis.py + analyze_results.py

make website           # rebuild website/public/data/*.json (must precede website-build)

make data              # = calibration + website



# Website

make website-dev       # vite dev server  → http://localhost:5173

make website-build     # static build     → website/dist/

make website-preview   # preview the production build

make website-deploy    # rsync website/dist/ to DEPLOY_HOST:DEPLOY_PATH

                       # override per invocation:

                       #   make website-deploy DEPLOY_HOST=user@host \

                       #                       DEPLOY_PATH=/var/www/research/ikp/



make all               # data → figs → pdf

```



For subpath deploys (e.g. `https://example.com/research/ikp/`), set

`BASE_URL=/research/ikp/ make website-build`. See `website/README.md` for full

website documentation, nginx config, and GitHub Pages instructions.



## Repo layout



```

ikp-paper/

├── README.md               ← this file

├── TOOLKIT.md              ← ikp_estimate.py reference

├── REPRODUCTION.md         ← figure/table ⇄ script map

├── ADVERSARIAL_IKP.md      ← how cheaply an operator can game its estimate

├── IKP_V2.md               ← gaming-resistant estimator (refusal interval + held-out split)

├── requirements.txt

│

├── paper/                  ← LaTeX sources

│   ├── main.tex  main.pdf  appendix.tex  references.bib

│   ├── research-plan.md    ← original planning document

│   └── figures/            ← PDF/PNG figures + generators (all main & appendix figs)

│       ├── generate_figures.py            (main-text figs 1–6, 8)

│       └── generate_appendix_figures.py   (appendix figs A1–A4)

│

├── configs/

│   ├── experiment.json     ← tier definitions, API settings, seeds

│   ├── models.json         ← calibration-set models (open, known size)

│   └── all_models.json     ← full roster (188 models evaluated)

│

├── data/                   ← see data/README.md for schemas

│   ├── probes/

│   │   ├── final_probe_set_v8.json  ← THE 1,400 probes (the benchmark)

│   │   ├── final_probe_set_clean.json ← the cleaned 1,311-probe subset (paper results)

│   │   ├── clean_mask.json          ← flags the 89 dropped probes; clean_ids

│   │   ├── researcher_probes.json   ← researcher sub-probe source

│   │   └── archive/                 ← earlier probe versions (v1..v7, batches, candidates)

│   ├── results/.json         ← per-model raw evaluations (188 files)

│   ├── results/evaluation_summary.json  ← aggregated, consumed by every figure

│   ├── calibration/calibration_fit.json ← fitted log-linear calibration

│   ├── researcher_citations.json        ← T4–T7 researcher metadata

│   ├── researcher_recognition_rates.json

│   ├── densing_analysis_data.csv        ← Densing-Law table (for Fig 8)

│   ├── notes/                           ← exploratory analysis markdown

│   └── archive/                         ← superseded runs (results_v7, …)

│

├── results/

│   ├── figures/archive/    ← early-draft plots (superseded by paper/figures/)

│   └── tables/             ← .tex tables \input'ed by the paper

│

├── scripts/                ← see scripts/README.md for a full index

│   ├── ikp_estimate.py     ← one-model estimator (public entrypoint)

│   ├── run_all_models.py   ← bulk evaluation across the full roster

│   ├── run_evaluation.py   ← single-model evaluator

│   ├── 01_..15_*.py        ← numbered dataset pipeline

│   ├── analyze_results.py, loo_cv_analysis.py, show_progress.py

│   └── legacy/             ← one-off / superseded dev scripts (kept for audit)

│

├── pipeline/               ← probe generation + calibration library

├── src/                    ← evaluation runtime (api_client, probe_runner, scorer, …)

├── cli/                    ← interactive reader CLI (research + eval modes)

└── website/                ← React companion site

```



All active scripts resolve paths via `Path(__file__).parent.parent`, so

they expect to live in `scripts/`. Scripts under `scripts/legacy/`

have been patched to three-`..` (`.parent.parent.parent`) and still

work when invoked directly.



## How it works (one paragraph)



Each probe is a short factual question with a gold answer, scored by a

Gemini 3 Flash Preview judge. Researcher subfield probes use a 4-way

evidence-aware judge (CORRECT_STRONG = subfield + verifiable evidence

item; CORRECT_WEAK = subfield only; REFUSAL; WRONG); other probes use

a 3-way judge (CORRECT / REFUSAL / WRONG). Penalized accuracy scores

each probe in `{+1.0, +0.5, 0, λ}` for the four classes with `λ = -1`

(WRONG); hallucinations are penalized to discourage guessing. The

calibration curve, loaded at runtime from

`data/results/calibration_refit_v2.json` (λ=0), is

`log10(params_B) = 6.701 · accuracy − 1.461` (equivalently the fitted

`accuracy = 0.149 · log10(params_B) + 0.218`; R² ≈ 0.91, no-penalty λ=0,

LOO median fold error ≈1.6×). For MoE models, *total* parameters

predict accuracy (R² = 0.79) much better than active parameters

(R² = 0.51) — so the curve is fit against total parameter count.



## Requirements



- Python ≥ 3.10

- An API key for the model(s) you want to evaluate (OpenRouter covers

  all 188 evaluated models; OpenAI-compatible endpoints also work)

- An `OPENROUTER_API_KEY` for the judge (always Gemini 3 Flash Preview)

- ~$0.10–$3 per model to score the full 1,400 probes, depending on the

  model priced at OpenRouter rates. Get the exact figure up-front with

  `python scripts/ikp_budget.py --model ` (no API key needed) — see

  the "Budgeting a run" section of `TOOLKIT.md`.



## Citing



```bibtex

@misc{li2026incompressibleknowledgeprobesestimating,

  title         = {Incompressible Knowledge Probes: Estimating Black-Box LLM

                   Parameter Counts via Factual Capacity},

  author        = {Bojie Li},

  year          = {2026},

  eprint        = {2604.24827},

  archivePrefix = {arXiv},

  primaryClass  = {cs.LG},

  url           = {https://arxiv.org/abs/2604.24827}

}

```



## License



Code: MIT. Probe set and per-model results: CC BY 4.0.