{"id":49891448,"url":"https://github.com/seryai/mdkit","last_synced_at":"2026-05-15T21:18:37.895Z","repository":{"id":354070024,"uuid":"1221999512","full_name":"seryai/mdkit","owner":"seryai","description":null,"archived":false,"fork":false,"pushed_at":"2026-04-27T05:29:22.000Z","size":108,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-27T07:17:13.713Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/seryai.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE-APACHE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-27T00:11:52.000Z","updated_at":"2026-04-27T05:40:20.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/seryai/mdkit","commit_stats":null,"previous_names":["mdkit-project/mdkit","seryai/mdkit"],"tags_count":14,"template":false,"template_full_name":null,"purl":"pkg:github/seryai/mdkit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seryai%2Fmdkit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seryai%2Fmdkit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seryai%2Fmdkit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seryai%2Fmdkit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/seryai","download_url":"https://codeload.github.com/seryai/mdkit/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seryai%2Fmdkit/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33080777,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-15T20:25:35.270Z","status":"ssl_error","status_checked_at":"2026-05-15T20:25:34.732Z","response_time":103,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-15T21:18:36.747Z","updated_at":"2026-05-15T21:18:37.874Z","avatar_url":"https://github.com/seryai.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"# mdkit\n\n**A Rust toolkit for getting markdown out of any document.** Built for\nTauri / Iced / native desktop apps that want best-in-class document\nextraction without a 350 MB Python sidecar.\n\n\u003e **Status:** v0.7 — **API stability candidate for 1.0**. Format\n\u003e coverage closed in v0.6 (PDF with scanned + mixed-content OCR,\n\u003e Pandoc-formats, spreadsheets, CSV/TSV, HTML, OCR backends for\n\u003e macOS / Windows / Linux). v0.7 freezes the public surface — see\n\u003e the [stability section](#stability-v07) below for what's locked\n\u003e in. v0.7.x will iterate on examples + cookbook docs. 1.0 ships\n\u003e once the API is exercised by at least one downstream production\n\u003e user. Watch / star the repo to follow along.\n\n## Why this exists\n\nMost Rust desktop apps that need to read DOCX / PDF / PPTX today have\ntwo choices, both bad:\n\n1. **Bundle a Python sidecar with [markitdown][markitdown]** — works\n   well, ~350 MB on disk, ~1 second cold-start per parse, single\n   process is GIL-locked. Fine for hobby projects, painful at scale.\n2. **Use [markitdown-rs][markitdown-rs]** — pure Rust, much smaller,\n   but PDF extraction is basic (no layout, no OCR) and DOCX support\n   drops headings, lists, hyperlinks, images.\n\n`mdkit` is the third option: **dispatch to the best tool per format,\nprefer in-process Rust crates and OS-native APIs, fall back to a\nsingle Pandoc binary for the formats Pandoc owns the gold standard\nfor.**\n\nThe composition we ship by default:\n\n| Format | Backend | Why |\n|---|---|---|\n| DOCX, PPTX, EPUB, RTF, ODT, LaTeX | [Pandoc][pandoc] sidecar | Best-in-world conversion quality. ~150 MB but you bundle it once. |\n| PDF (text) | [`pdfium-render`][pdfium] | Google's Pdfium engine, in-process, layout-aware. ~5 MB. |\n| PDF (scanned + mixed-content) | Pdfium renders pages → platform OCR (Vision / Windows.Media.Ocr) | Per-page composition — pages with text pass through pdfium, empty pages get OCR'd. Auto-wired by `Engine::with_defaults` when both `pdf` and `ocr-platform` features are on. |\n| Standalone images | Platform-native OCR — Vision.framework on macOS, Windows.Media.Ocr on Windows, ONNX-based ([Surya][surya]) on Linux | OS-quality on Mac/Win for free. ONNX models on Linux. |\n| XLSX, XLS, ODS | [`calamine`][calamine] | Already the Rust ecosystem standard. |\n| CSV, TSV | [`csv`][csv] | Stdlib-quality. |\n| HTML | [`html2md`][html2md] (or Pandoc, configurable) | Default cheap, optional best. |\n| Jupyter (IPYNB) | Built-in via `serde_json` | Pure-Rust JSON parse, no external deps. ~50 lines. |\n\nTotal binary size with all backends: **~50-200 MB** depending on\nwhich optional features you enable, vs ~350 MB for a Python\nmarkitdown sidecar.\n\n## Design principles\n\n1. **Best output per format, not uniform mediocrity.** A single Rust\n   crate that handles 20 formats poorly is worse than a dispatcher\n   that uses Pandoc for what Pandoc is best at and Pdfium for what\n   Pdfium is best at.\n2. **OS-native first.** macOS PDFKit + Vision.framework, Windows\n   Windows.Data.Pdf + Windows.Media.Ocr — these are battle-tested\n   parsers Apple and Microsoft already paid for. We use them.\n3. **In-process where possible, sidecar where necessary.** Process\n   spawn is ~50-100 ms per file. For a folder of 1,000 files, that's\n   real time. Pandoc is the one sidecar we accept; everything else is\n   a Rust crate or OS-native FFI.\n4. **Privacy-respecting.** Every extractor runs entirely on-device.\n   No telemetry, no cloud round-trips, no analytics. (LLM-based image\n   description is opt-in and uses the caller's own provider key.)\n5. **Graceful degradation.** A bad PDF doesn't crash the process; it\n   returns a typed error. Missing optional dependencies don't break\n   the build; they disable specific extractors via feature flags.\n6. **Small, stable, public surface.** The `Extractor` trait + `Engine`\n   dispatcher are the API. Backends are implementation details that\n   can be swapped without breaking callers.\n\n## Quick start\n\n```rust\nuse mdkit::Engine;\nuse std::path::Path;\n\nlet engine = Engine::with_defaults();\nlet doc = engine.extract(Path::new(\"report.pdf\"))?;\nprintln!(\"{}\", doc.markdown);\n```\n\nTo register your own extractor for a custom format:\n\n```rust\nuse mdkit::{Engine, Extractor, Document, Result};\nuse std::path::Path;\n\nstruct MyParser;\n\nimpl Extractor for MyParser {\n    fn extensions(\u0026self) -\u003e \u0026[\u0026'static str] { \u0026[\"custom\"] }\n    fn extract(\u0026self, path: \u0026Path) -\u003e Result\u003cDocument\u003e {\n        Ok(Document::new(std::fs::read_to_string(path)?))\n    }\n}\n\nlet mut engine = Engine::new();\nengine.register(Box::new(MyParser));\n```\n\n## Feature flags\n\n`mdkit` ships with backends behind feature flags so you only pay for\nwhat you use:\n\n```toml\n[dependencies]\nmdkit = { version = \"0.1\", features = [\"pdf\", \"pandoc\", \"ocr-platform\", \"calamine\"] }\n```\n\n| Feature | Adds | Approx. size cost |\n|---|---|---|\n| `pdf` | `pdfium-render` for PDF text extraction | ~5 MB |\n| `pandoc` | Pandoc sidecar wrapper for DOCX/PPTX/EPUB/RTF/ODT/LaTeX | ~150 MB sidecar to bundle separately |\n| `ocr-platform` | macOS Vision.framework + Windows.Media.Ocr (Linux falls back to `ocr-onnx`) | 0 on macOS/Win |\n| `ocr-onnx` | ONNX-runtime PaddleOCR via [`oar-ocr`][oar-ocr] — works on all platforms incl. Linux. Caller supplies model files (~12 MB total) and `libonnxruntime`. | ~30 MB compiled |\n| `ocr-onnx-download` | Adds `ocr-onnx` and lets `oar-ocr` fetch ONNX Runtime native libs at build / first use | 0 (downloads at build) |\n| `calamine` | XLSX / XLS / ODS via `calamine` | ~1 MB |\n| `csv` | CSV / TSV | \u003c1 MB |\n| `html` | HTML via `html2md` | \u003c1 MB |\n| `ipynb` | Jupyter notebooks via `serde_json` | \u003c1 MB |\n| `default` | `pdf`, `calamine`, `csv`, `html`, `ipynb` (the in-process Rust ones) | ~7 MB |\n\nNot enabling `pandoc` or `ocr-platform` is fine — extractors for those\nformats simply won't be registered, and `Engine::extract` will return\n`Error::UnsupportedFormat` for them.\n\n## Examples\n\nRunnable example programs live in [`examples/`](examples/):\n\n- [`extract.rs`](examples/extract.rs) — print the extracted markdown\n  for any document. Surfaces backend registration failures\n  (libpdfium / pandoc / etc.) at startup. Run with:\n  ```bash\n  cargo run --example extract -- /path/to/document.pdf\n  cargo run --example extract --features pandoc -- /path/to/report.docx\n  cargo run --example extract --features ocr-platform -- /path/to/scan.pdf\n  ```\n- [`batch.rs`](examples/batch.rs) — non-recursive folder → `.md`\n  batch converter. Run with:\n  ```bash\n  cargo run --example batch -- /path/to/folder /path/to/output\n  ```\n- [`custom_extractor.rs`](examples/custom_extractor.rs) —\n  implements the `Extractor` trait for a custom file format,\n  showing the registration pattern. Run with:\n  ```bash\n  cargo run --example custom_extractor -- /path/to/file.rot\n  ```\n\n## License\n\nDual-licensed under [MIT](LICENSE-MIT) OR [Apache 2.0](LICENSE-APACHE)\nat your option. SPDX: `MIT OR Apache-2.0`.\n\n## Stability (v0.7+) {#stability-v07}\n\nv0.7 is the **API stability candidate** for 1.0. The following surface\nis committed to and will only change with a major version bump:\n\n- The `Extractor` trait shape — required methods, default implementations, `Send + Sync` bound.\n- `Engine` construction + dispatch — `new`, `with_defaults`, `with_defaults_diagnostic`, `register`, `extract`, `extract_bytes`, `len`, `is_empty`.\n- `Document` field set + `Document::new`. Marked `#[non_exhaustive]` so we can add fields (page count, language, confidence) without major bumps.\n- `Error` enum semantics. Marked `#[non_exhaustive]` so we can add variants without major bumps. **Pattern-matchers must include a wildcard arm.**\n- Feature flag names: `pdf`, `pandoc`, `calamine`, `csv`, `html`, `ocr-platform`, `ocr-onnx`, `ocr-onnx-download`, `full`.\n- Backend `name()` strings — used by callers for filtering / logging.\n\nThe following are **implementation details** and may change in minor versions:\n\n- Internal layout of any specific extractor (private fields, helper methods).\n- Exact set of `Document.metadata` keys per backend (new keys may appear; documented keys stay).\n- Auto-registration order in `Engine::with_defaults` (when multiple backends claim overlapping extensions; documented priority stays).\n- Internal sidecar / FFI details (Pandoc's `--server` mode, ONNX runtime version, libpdfium binding).\n\n1.0 will be cut once the API is exercised by at least one downstream production user.\n\n## Status \u0026 roadmap\n\nThis is a young project. v0.1 ships the trait surface, dispatch\nengine, and a no-op test extractor. Real backends land per the\nroadmap below:\n\n- [x] **v0.2 — `pdf` feature (`pdfium-render` integration).** `PdfiumExtractor`\n      registers automatically in `Engine::with_defaults()`; falls back\n      gracefully when libpdfium isn't installed. See `src/pdf.rs` for\n      libpdfium installation notes.\n- [x] **v0.3 — `pandoc` feature.** `PandocExtractor` covers DOCX, PPTX,\n      EPUB, RTF, ODT, LaTeX, HTML via the `pandoc` binary. Auto-\n      registers when `pandoc` is on PATH; supports\n      `with_binary(absolute_path)` for shipping pandoc next to your\n      app. CHANGELOG.md added.\n- [x] **v0.4 — `calamine` + `csv` + `html` features (all in-process).**\n      `CalamineExtractor` (XLSX/XLS/XLSB/XLSM/ODS), `CsvExtractor`\n      (CSV/TSV with auto-delimiter), `Html2mdExtractor` (HTML/HTM,\n      registered before Pandoc so it wins by default for HTML).\n      Engine registration order documented inline.\n- [x] **v0.5 — `ocr-platform` feature (macOS Vision + Windows.Media.Ocr).**\n      `VisionOcrExtractor` (macOS) handles PNG / JPG / TIFF / BMP /\n      GIF / HEIC via Apple Vision (neural-network-based, Apple Neural\n      Engine-accelerated on Apple Silicon). `WindowsOcrExtractor`\n      (Windows) handles PNG / JPG / TIFF / BMP / GIF via the OS-built-in\n      Windows.Media.Ocr engine. v0.5.3 adds **scanned-PDF → OCR\n      composition**: `PdfiumExtractor` accepts an OCR fallback at\n      construction; when text extraction yields empty markdown, each\n      page is rendered to a PNG and routed through OCR with `## Page N`\n      headings. Linux ONNX backend lands in v0.6.\n- [x] **v0.6 — `ocr-onnx` feature (PaddleOCR via [`oar-ocr`][oar-ocr]).**\n      `OnnxOcrExtractor` wraps PaddleOCR's detection + recognition\n      ONNX models through `ort`; works on Linux + macOS + Windows +\n      WebAssembly. Caller supplies the three model files\n      (det + rec + dict, ~12 MB total English-only) — download from\n      \u003chttps://github.com/GreatV/oar-ocr/releases\u003e. Opt into\n      `ocr-onnx-download` to let oar-ocr fetch the ONNX Runtime\n      native lib for you.\n- [x] **v0.7 — API stability candidate (`#[non_exhaustive]` audit, `#[must_use]` audit, stability commitments doc).**\n      v0.7.0 freezes the public surface for 1.0. v0.7.x will iterate\n      on examples + cookbook docs + niche backend polish that doesn't\n      change the public API. 1.0 ships once the API is exercised by\n      at least one downstream production user.\n\nIssues, PRs, and design discussion welcome at\n\u003chttps://github.com/seryai/mdkit/issues\u003e.\n\n## Used by\n\n`mdkit` was extracted from the document-extraction layer of [Sery\nLink][sery], a privacy-respecting data network for the files on your\nmachines. If you use `mdkit` in your project, please open a PR to\nadd yourself here.\n\n## Acknowledgements\n\n`mdkit` would not exist without:\n\n- [markitdown][markitdown] — Microsoft's Python implementation, the\n  prior art and quality benchmark for \"any-doc-to-markdown.\"\n- [markitdown-rs][markitdown-rs] — `uhobnil`'s Rust port, which\n  proved the Rust-side feasibility and inspired the dispatch design.\n- [Pandoc][pandoc] — John MacFarlane's universal document converter,\n  the standard the academic publishing world is built on.\n- [Pdfium][pdfium] — Google's PDF engine, free for everyone to use.\n- [calamine][calamine] — `tafia`'s industry-standard Rust XLSX parser.\n\n[markitdown]: https://github.com/microsoft/markitdown\n[markitdown-rs]: https://github.com/uhobnil/markitdown-rs\n[oar-ocr]: https://github.com/GreatV/oar-ocr\n[pandoc]: https://pandoc.org\n[pdfium]: https://pdfium.googlesource.com/pdfium/\n[calamine]: https://github.com/tafia/calamine\n[csv]: https://crates.io/crates/csv\n[html2md]: https://crates.io/crates/html2md\n[surya]: https://github.com/VikParuchuri/surya\n[sery]: https://sery.ai\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fseryai%2Fmdkit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fseryai%2Fmdkit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fseryai%2Fmdkit/lists"}