{"id":51488597,"url":"https://github.com/badrusiddique/tokendiet","last_synced_at":"2026-07-07T09:01:37.983Z","repository":{"id":366586212,"uuid":"1276911371","full_name":"badrusiddique/tokendiet","owner":"badrusiddique","description":"Convert PDFs to lean Markdown so Claude reads the cheap version — with a measured token and cost savings report. A Claude skill plus CLI.","archived":false,"fork":false,"pushed_at":"2026-06-22T12:21:40.000Z","size":148,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-22T14:15:44.104Z","etag":null,"topics":["anthropic","claude","claude-skill","llm","llm-tools","markdown","pdf-to-markdown","token-optimization","tokens"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/badrusiddique.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-22T12:07:49.000Z","updated_at":"2026-06-22T12:19:54.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/badrusiddique/tokendiet","commit_stats":null,"previous_names":["badrusiddique/tokendiet"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/badrusiddique/tokendiet","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/badrusiddique%2Ftokendiet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/badrusiddique%2Ftokendiet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/badrusiddique%2Ftokendiet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/badrusiddique%2Ftokendiet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/badrusiddique","download_url":"https://codeload.github.com/badrusiddique/tokendiet/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/badrusiddique%2Ftokendiet/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35221308,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-07T02:00:07.222Z","response_time":90,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anthropic","claude","claude-skill","llm","llm-tools","markdown","pdf-to-markdown","token-optimization","tokens"],"created_at":"2026-07-07T09:01:34.895Z","updated_at":"2026-07-07T09:01:37.919Z","avatar_url":"https://github.com/badrusiddique.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003eTokendiet\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cb\u003eConvert PDFs to lean Markdown so Claude reads the cheap version — and see exactly how many tokens (and dollars) you saved.\u003c/b\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#quickstart\"\u003eQuickstart\u003c/a\u003e ·\n  \u003ca href=\"#how-it-works\"\u003eHow it works\u003c/a\u003e ·\n  \u003ca href=\"#benchmarks\"\u003eBenchmarks\u003c/a\u003e ·\n  \u003ca href=\"#use-as-a-claude-skill\"\u003eClaude skill\u003c/a\u003e ·\n  \u003ca href=\"#faq\"\u003eFAQ\u003c/a\u003e\n\u003c/p\u003e\n\n---\n\n## Why\n\nWhen you hand Claude a PDF, it doesn't just read the text — it **also renders every page to an image and tokenizes that image**, on top of the extracted text. For a text-heavy document, those page images are pure cost: the same words as clean Markdown, at a multiple of the tokens.\n\n**Tokendiet** converts the document to Markdown first, so Claude reads the text-only version. Then — and this is the part other converters skip — it **reports the measured token-$ you saved**, so the win is visible, not hand-waved.\n\n```\n$ tokendiet convert paper.pdf\nWrote paper.md\nTokendiet — paper.pdf\n  Pages: 15\n  Native PDF (estimate): 48,830 tokens (10,820 text + 38,010 image)\n  Markdown (estimate):   10,820 tokens\n  Saved: 38,010 tokens (78%)\n  Saved per call: Claude Opus 4.8: $0.1900, Claude Sonnet 4.6: $0.1140, Claude Haiku 4.5: $0.0380\n```\n\n## Quickstart\n\n```bash\n# 1. Install (from source for now; PyPI release coming)\ngit clone https://github.com/badrusiddique/tokendiet\ncd tokendiet\nuv tool install .          # or: pipx install .   /   pip install .\n\n# 2. Convert — PDF, HTML, a URL, or an image\ntokendiet convert report.pdf                       # writes report.md + prints savings\ntokendiet convert page.html --report json          # machine-readable report\ntokendiet convert https://en.wikipedia.org/wiki/Markdown   # fetch + convert a web page\ntokendiet convert scan.png                         # OCR an image (needs the ocr extra)\ntokendiet convert report.pdf --stdout --quiet \u003e report.md\n```\n\nFor image OCR, install the optional engine: `pip install 'tokendiet[ocr]'` (CPU-only, models bundled). Otherwise — no GPU, no cloud API, no account.\n\n## Benchmarks\n\nHonest, reproducible, run on your own machine — **no borrowed marketing numbers.** Estimates use `tiktoken` as a proxy for Claude's tokenizer plus Anthropic's documented image-token heuristic.\n\n| Document | Type | Native (tok) | Markdown (tok) | Saved | % |\n|---|---|--:|--:|--:|--:|\n| Real research paper (arXiv 1706.03762) | PDF, 15p | 48,830 | 10,820 | 38,010 | **78%** |\n| Real Wikipedia article (live HTML)† | HTML | ~64,200 | ~9,500 | ~54,700 | **~85%** |\n| Prose (public-domain text) | PDF, 4p | 12,703 | 3,435 | 9,268 | **73%** |\n| Bloated article (inline CSS/JS) | HTML | 6,643 | 1,897 | 4,746 | **71%** |\n| Report with table | PDF, 2p | 5,687 | 1,053 | 4,634 | **81%** |\n| **Total** | | **~138,100** | **~26,700** | **~111,400** | **~81%** |\n\n\u003csub\u003e† The Wikipedia row is a **live** fetch, so its exact tokens (and the totals) drift a little between runs — shown rounded. The deterministic rows are exact. Run `python benchmarks/run.py` for current numbers.\u003c/sub\u003e\n\nReproduce:\n\n```bash\npython benchmarks/make_corpus.py    # generates docs + downloads the real paper \u0026 web page\npython benchmarks/run.py            # writes benchmarks/RESULTS.md\n```\n\nFor **PDF** the savings come from eliminating per-page image tokens; for **HTML** from shedding markup (tags, attributes, inline scripts/styles). Both are the real mechanism, not an artifact. See [`benchmarks/RESULTS.md`](benchmarks/RESULTS.md) for the full method and caveats.\n\n**Images** (with the `ocr` extra): sending a page as an image costs ~2,317 tokens; OCR'd Markdown is ~665 — **~71% saved**, verified with a real OCR engine. **Why Markdown and not base64/LaTeX/plain text?** We measured — base64 is 7–360× *worse*; Markdown is the right container. Details: [docs/format-support.md](docs/format-support.md#why-markdown-not-base64-binary-latex-).\n\n## How it works\n\n```\n                 ┌──────────────────────────────────────────┐\n   your.pdf  ──▶ │  tokendiet convert                        │ ──▶  your.md\n                 │  • PyMuPDF extracts structured text        │      (text only)\n                 │  • emits clean Markdown                     │\n                 │  • estimates native tokens (text + image)  │ ──▶  savings report\n                 │  • estimates Markdown tokens               │      (tokens + $)\n                 └──────────────────────────────────────────┘\n```\n\nClaude then reads `your.md` instead of `your.pdf`, paying for text only. **HTML files and URLs** follow the same path: Tokendiet extracts the main content (via `trafilatura`, falling back to a full strip-and-convert when no article body is found) and emits clean Markdown — shedding nav, sidebars, footers, scripts, and styles. The native baseline there is the raw markup, not page images. **Images** (with the `ocr` extra) are OCR'd to text — replacing the image tokens Claude would pay to *see* the page. Different input, same destination: lean Markdown.\n\n## Use as a Claude skill\n\nTokendiet ships a [`SKILL.md`](SKILL.md). Once installed (see [docs/getting-started.md](docs/getting-started.md)), when you reference a PDF or web page in Claude Code, Claude converts it with Tokendiet, reads the lean Markdown, and shows you the savings — instead of burning tokens on the native document.\n\n\u003e **A note on honesty:** a skill triggers on *intent*, not on a file-drop event. Tokendiet is **convert-on-reference**, not a magic upload interceptor. No false promises.\n\n## When *not* to use it\n\nTokendiet is the wrong tool when the **visuals** are the point:\n\n- Charts, diagrams, infographics, scanned documents, complex multi-column layouts, or math rendered as images — Markdown drops them. Let Claude read the PDF natively.\n- Tokendiet detects image-only/scanned PDFs, no-text images, and **low-confidence OCR**, and **warns you** instead of silently producing empty or wrong Markdown.\n\n**Does it lose your content?** We measure it ([`benchmarks/fidelity_check.py`](benchmarks/fidelity_check.py)): PDF keeps **95–98%** of the text (tables included), HTML keeps the full article body (only chrome is dropped), image OCR recovers **~90–95%** (weaker on small text/numbers — hence the warning). Body *text* is never silently dropped; only genuinely-visual content can't survive a text conversion. Full breakdown: [docs/format-support.md](docs/format-support.md#fidelity--whats-preserved-whats-lost).\n\n## Comparison\n\n| | Tokendiet | [MarkItDown](https://github.com/microsoft/markitdown) | Plain PDF→MD skills |\n|---|---|---|---|\n| PDF / HTML / URL / image → Markdown | ✅ | ✅ / ✅ / ❌ / ✅ | PDF only |\n| **Measured token-$ savings report** | ✅ | ❌ | ❌ |\n| Reproducible benchmark in-repo | ✅ | ❌ | ❌ |\n| Scanned/visual-loss warnings | ✅ | partial | ❌ |\n| Dependency-light (no GPU/cloud) | ✅ | ✅ | varies |\n\nTokendiet isn't trying to out-parse MarkItDown — it's the one that **shows you the win**.\n\n## FAQ\n\n**Does this really save tokens?** Yes, for text-heavy documents — by removing the per-page image tokens Claude pays on the native path. The benchmark numbers above are reproducible on your machine.\n\n**Are the numbers exact?** They're labelled **estimates** (tiktoken proxy + documented image heuristic). For exact figures, validate against Anthropic's `count_tokens` API; the offline estimate is designed to be close and conservative.\n\n**What about DOCX / PPTX / XLSX?** **Deliberately not supported** — we measured, and they don't save tokens. Office files have no expensive native form (Claude doesn't image them like PDFs, and their text extraction is already Markdown-sized), so converting them is equal or *larger*. We won't ship a feature that doesn't deliver on the promise. Full data and the reproducible probe: [docs/format-support.md](docs/format-support.md). For convenience conversion, use [MarkItDown](https://github.com/microsoft/markitdown).\n\n**Roadmap?** PDF (v0.1 ✓) → HTML + URLs (v0.2 ✓) → Office investigated \u0026 excluded with data (v0.3 ✓) → main-content extraction, 85% on Wikipedia (v0.4 ✓) → images via OCR, 71% verified (v0.5 ✓) → video → transcript (the 567× win, needs ASR). See [docs/ROADMAP.md](docs/ROADMAP.md).\n\n## License\n\n[MIT](LICENSE) © Badru Siddique\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbadrusiddique%2Ftokendiet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbadrusiddique%2Ftokendiet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbadrusiddique%2Ftokendiet/lists"}